Oct 31 16:01:49.112971 ip-10-0-142-127 systemd[1]: Starting Kubernetes Kubelet... Oct 31 16:01:49.515439 ip-10-0-142-127 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 16:01:49.515439 ip-10-0-142-127 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 16:01:49.515439 ip-10-0-142-127 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 16:01:49.515439 ip-10-0-142-127 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 16:01:49.519953 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519860 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.519953 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519951 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.519953 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519955 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.519953 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519958 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519962 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519965 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519969 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519971 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519974 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519977 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519980 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519983 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519985 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519988 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519992 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519994 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519997 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.519999 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520002 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520005 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520007 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520010 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520013 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.520085 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520015 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520018 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520021 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520024 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520026 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520029 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520031 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520044 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520048 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520053 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520057 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520060 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520064 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520067 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520070 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520073 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520076 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520079 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.520545 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520081 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520085 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520088 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520090 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520093 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520096 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520099 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520102 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520105 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520107 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520110 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520113 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520116 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520118 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520121 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520123 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520126 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520129 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520131 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520134 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.520986 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520137 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520139 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520142 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520144 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520150 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520153 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520156 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520159 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520162 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520165 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520167 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520170 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520173 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520176 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520179 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520181 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520184 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520187 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520189 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520192 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.521495 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520194 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520198 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520200 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520203 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520205 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520208 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520211 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520213 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520216 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520218 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520221 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520223 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520226 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520228 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520231 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520233 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520236 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520238 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520242 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520245 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.521997 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520248 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520251 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520253 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520256 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.520259 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521092 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521100 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521104 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521106 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521109 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521112 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521115 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521118 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521121 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521124 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521127 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521129 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521133 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521136 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521138 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.522504 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521141 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521143 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521145 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521148 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521151 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521154 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521156 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521159 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521161 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521164 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521168 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521170 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521174 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521177 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521180 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521182 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521185 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521187 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521190 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.523010 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521192 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521195 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521197 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521200 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521202 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521205 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521207 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521210 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521213 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521216 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521218 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521221 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521223 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521226 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521229 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521231 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521234 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521237 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521240 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.523463 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521242 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521245 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521247 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521250 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521253 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521256 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521259 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521262 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521265 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521267 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521270 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521272 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521275 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521279 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521283 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521285 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521288 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521290 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521293 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.523931 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521295 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521298 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521302 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521305 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521307 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521310 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521312 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521315 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521317 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521320 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521322 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521325 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521328 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521330 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521333 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521336 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521339 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521341 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521344 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521349 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.524383 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521353 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521356 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521359 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521361 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521364 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521367 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521370 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521373 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521376 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521378 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521381 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521384 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521386 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521389 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521943 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521949 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521952 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521955 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521958 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521961 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.524886 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521964 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521969 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521973 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521976 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521979 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521982 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521985 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521987 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521990 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521993 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521996 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.521999 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522002 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522004 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522007 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522010 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522013 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522015 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522018 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.525379 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522020 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522023 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522026 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522028 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522030 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522033 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522035 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522038 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522042 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522045 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522048 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522051 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522054 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522056 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522059 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522061 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522063 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522067 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522069 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522072 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.525864 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522074 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522077 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522080 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522083 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522085 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522088 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522091 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522093 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522096 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522099 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522101 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522104 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522107 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522110 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522114 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522117 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522120 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522122 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522125 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.526327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522128 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522131 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522134 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522136 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522139 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522141 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522144 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522146 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522149 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522151 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522154 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522156 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522159 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522161 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522165 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522168 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522170 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522173 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522176 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522178 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.526834 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522181 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522183 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522186 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522189 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522191 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522194 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522196 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522199 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522201 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522204 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522206 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522209 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522211 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522213 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522216 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522219 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522222 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522224 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522228 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522230 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.527323 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522233 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.522235 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522342 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522350 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522357 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522362 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522367 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522371 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522375 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522380 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522383 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522386 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522390 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522393 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522396 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522399 2573 flags.go:64] FLAG: --cgroup-root="" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522403 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522406 2573 flags.go:64] FLAG: --client-ca-file="" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522409 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522412 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522416 2573 flags.go:64] FLAG: --cluster-domain="" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522419 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522422 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522425 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 16:01:49.527800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522428 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522433 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522436 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522439 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522443 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522447 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522450 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522453 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522456 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522459 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522463 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522466 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522472 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522475 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522478 2573 flags.go:64] FLAG: --enable-server="true" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522482 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522486 2573 flags.go:64] FLAG: --event-burst="100" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522489 2573 flags.go:64] FLAG: --event-qps="50" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522492 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522496 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522499 2573 flags.go:64] FLAG: --eviction-hard="" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522503 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522507 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522510 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522513 2573 flags.go:64] FLAG: --eviction-soft="" Oct 31 16:01:49.528392 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522516 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522519 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522522 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522526 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522529 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522532 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522535 2573 flags.go:64] FLAG: --feature-gates="" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522539 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522543 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522546 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522550 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522553 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522557 2573 flags.go:64] FLAG: --help="false" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522560 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522563 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522566 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522569 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522573 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522576 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522580 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522584 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522587 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522590 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 16:01:49.529165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522593 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522596 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522599 2573 flags.go:64] FLAG: --kube-reserved="" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522602 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522605 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522608 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522611 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522614 2573 flags.go:64] FLAG: --lock-file="" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522617 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522620 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522623 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522629 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522632 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522635 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522638 2573 flags.go:64] FLAG: --logging-format="text" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522641 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522661 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522667 2573 flags.go:64] FLAG: --manifest-url="" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522671 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522675 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522679 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522683 2573 flags.go:64] FLAG: --max-pods="110" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522687 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522690 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522693 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 16:01:49.529793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522696 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522699 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522702 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522707 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522715 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522718 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522721 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522724 2573 flags.go:64] FLAG: --pod-cidr="" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522727 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522730 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522733 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522736 2573 flags.go:64] FLAG: --port="10250" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522739 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522742 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a53437074682e059" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522746 2573 flags.go:64] FLAG: --qos-reserved="" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522749 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522752 2573 flags.go:64] FLAG: --register-node="true" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522755 2573 flags.go:64] FLAG: --register-with-taints="" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522759 2573 flags.go:64] FLAG: --registry-burst="10" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522762 2573 flags.go:64] FLAG: --registry-qps="5" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522765 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522768 2573 flags.go:64] FLAG: --reserved-memory="" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522772 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522776 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522780 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522783 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 16:01:49.530381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522786 2573 flags.go:64] FLAG: --runonce="false" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522790 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522793 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522796 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522800 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522803 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522806 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522809 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522813 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522818 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522822 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522824 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522827 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522831 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522834 2573 flags.go:64] FLAG: --system-cgroups="" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522837 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522842 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522845 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522848 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522852 2573 flags.go:64] FLAG: --tls-min-version="" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522855 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522858 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522861 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522865 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522868 2573 flags.go:64] FLAG: --v="2" Oct 31 16:01:49.531013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522874 2573 flags.go:64] FLAG: --version="false" Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522878 2573 flags.go:64] FLAG: --vmodule="" Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522882 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.522886 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523021 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523026 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523030 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523032 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523035 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523038 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523041 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523043 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523046 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523049 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523052 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523054 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523058 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523061 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523064 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.531657 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523066 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523069 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523071 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523074 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523076 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523079 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523082 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523085 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523089 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523093 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523097 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523099 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523102 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523105 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523107 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523110 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523112 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523115 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523119 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523121 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.532119 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523124 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523126 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523129 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523132 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523134 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523137 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523140 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523143 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523145 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523149 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523152 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523154 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523157 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523159 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523162 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523165 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523167 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523170 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523173 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.532602 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523175 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523177 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523180 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523183 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523185 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523188 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523190 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523193 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523195 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523198 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523200 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523204 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523207 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523210 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523212 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523215 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523219 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523222 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523225 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.533073 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523229 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523232 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523235 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523238 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523241 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523244 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523247 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523249 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523252 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523255 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523257 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523260 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523262 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523265 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523267 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523270 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523272 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523275 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523278 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523281 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.533536 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523283 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523286 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523289 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523291 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523295 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523298 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523301 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523303 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523306 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523308 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523311 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523313 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523316 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.523341 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523379 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523384 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.534108 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523388 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523391 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523394 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523397 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523399 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523402 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523405 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523408 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523411 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523413 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523415 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523419 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523421 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523424 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523427 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523429 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523432 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523435 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523438 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.534493 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523440 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523444 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523447 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523450 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523452 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523455 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523458 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523460 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523463 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523466 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523470 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523474 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523477 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523482 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523485 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523487 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523490 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523492 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523495 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.535005 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523497 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523500 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523503 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523505 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523508 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523511 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523513 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523516 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523518 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523521 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523523 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523526 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523528 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523531 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523537 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523539 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523542 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523544 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523547 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.535482 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523550 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523553 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523555 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523558 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523560 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523563 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523565 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523569 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523572 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523575 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523577 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523580 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523582 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523585 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523588 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523590 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523593 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523596 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523598 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.536202 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523602 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523606 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523609 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523612 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523614 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523617 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523619 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523622 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523627 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523630 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523632 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523634 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523637 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523640 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523655 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523659 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523663 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523666 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523669 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523672 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.536872 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523676 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523684 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523687 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523689 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523693 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523696 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523698 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.523701 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.530603 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.530625 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.530658 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.530666 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530730 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530736 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530740 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530742 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530746 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530749 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530752 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.537345 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530755 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530757 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530760 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530763 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530766 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530769 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530771 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530774 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530777 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530779 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530782 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530785 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530787 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530790 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530792 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530796 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530798 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530801 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530803 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.537867 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530806 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530809 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530812 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530824 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530827 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530830 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530832 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530835 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530838 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530840 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530843 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530846 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530848 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530851 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530853 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530856 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530859 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530861 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530864 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530866 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.538304 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530869 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530871 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530874 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530877 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530880 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530883 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530885 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530888 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530891 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530893 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530896 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530899 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530901 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530904 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530907 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530910 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530918 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530922 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530924 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530927 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.538851 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530930 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530932 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530935 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530937 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530940 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530944 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530948 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530951 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530954 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530957 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530959 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530962 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530965 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530968 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530970 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530973 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530976 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530978 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530981 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.539327 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530983 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530986 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530989 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530991 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530994 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530997 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.530999 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531002 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531005 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531007 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531016 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531020 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531024 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531029 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531032 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531035 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531039 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531043 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531047 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.539789 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531049 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531053 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.531083 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531311 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531316 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531319 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531322 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531325 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531327 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531330 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531333 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531336 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531338 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531341 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531343 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531346 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.540237 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531349 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531352 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531355 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531357 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531360 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531362 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531365 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531367 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531377 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531380 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531383 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531386 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531388 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531391 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531394 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531397 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531400 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531402 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531405 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.540619 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531407 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531410 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531414 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531417 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531420 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531423 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531425 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531428 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531431 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531433 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531436 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531438 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531441 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531443 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531447 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531449 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531452 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531454 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531457 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531460 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.541083 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531462 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531465 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531473 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531477 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531482 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531486 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531489 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531493 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531498 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531502 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531505 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531508 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531511 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531514 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531516 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531519 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531521 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531524 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531527 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531529 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.541568 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531532 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531535 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531537 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531547 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531550 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531552 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531555 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531557 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531559 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531562 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531565 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531567 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531569 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531572 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531574 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531583 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531586 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531588 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531591 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531594 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.542120 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531597 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531601 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531605 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531608 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531611 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531613 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531616 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531618 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531621 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531623 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531626 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531628 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531631 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531634 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.531671 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531711 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:01:49.542578 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531715 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531718 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531721 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531723 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531726 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531729 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531731 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531734 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531736 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531739 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531743 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531747 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531756 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531759 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531762 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531765 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531767 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531770 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531772 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531775 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:01:49.542958 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531778 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531780 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531783 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531785 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531788 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531791 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531793 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531796 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531798 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531800 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531803 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531805 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531808 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531810 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531813 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531815 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531818 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531821 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531823 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:01:49.543432 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531826 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531828 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531831 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531833 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531836 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531838 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531846 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531849 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531852 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531855 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531857 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531860 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531863 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531866 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531868 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531870 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531873 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531876 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:01:49.543898 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531880 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531883 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531886 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531889 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531891 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531894 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531897 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531899 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531902 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531904 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531907 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531909 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531912 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531914 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531917 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531919 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531922 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531924 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531926 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531929 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:01:49.544338 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531932 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531942 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531945 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531947 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531950 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531953 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531955 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531958 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531960 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531963 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531965 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531968 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531970 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531973 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531975 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531978 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531981 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531983 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531985 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:01:49.544856 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531988 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531990 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531994 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531997 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.531999 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.532001 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.532004 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.532006 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:49.532009 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.532079 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.534320 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.537108 2573 server.go:1014] "Starting client certificate rotation" Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.537208 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.537238 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.539974 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 16:01:49.545377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.541285 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 16:01:49.550187 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.550167 2573 log.go:25] "Validated CRI v1 runtime API" Oct 31 16:01:49.554346 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.554329 2573 log.go:25] "Validated CRI v1 image API" Oct 31 16:01:49.555359 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.555330 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 16:01:49.558358 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.558337 2573 fs.go:135] Filesystem UUIDs: map[20011175-f2a4-4c46-9bd1-9b4d239776e9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e3fc3432-3a15-4341-a7c4-9ddea9cbe2cf:/dev/nvme0n1p3] Oct 31 16:01:49.558423 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.558357 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 16:01:49.564982 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.564851 2573 manager.go:217] Machine: {Timestamp:2026-10-31 16:01:49.563413407 +0000 UTC m=+0.333367609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093457 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26b5f2cef135b6e421564abb14eda7 SystemUUID:ec26b5f2-cef1-35b6-e421-564abb14eda7 BootID:34639023-6b40-4003-a846-8c34e47aaaca Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cc:28:1e:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cc:28:1e:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:dc:13:2e:2b:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 16:01:49.565598 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.565585 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 16:01:49.565768 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.565754 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 16:01:49.568923 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.568883 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 16:01:49.569069 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.568923 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-127.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 16:01:49.569682 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.569670 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 16:01:49.569725 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.569684 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 16:01:49.569725 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.569717 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 16:01:49.569779 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.569732 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 16:01:49.570803 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.570791 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 16:01:49.574166 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.574153 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 16:01:49.574297 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.574287 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 16:01:49.575154 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.575144 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 31 16:01:49.575188 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.575160 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 16:01:49.575188 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.575171 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 16:01:49.575188 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.575181 2573 kubelet.go:405] "Adding apiserver pod source" Oct 31 16:01:49.575293 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.575191 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 16:01:49.575707 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.575693 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 16:01:49.580356 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.580331 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 16:01:49.629096 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.629064 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 16:01:49.630196 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630180 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 16:01:49.630246 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630214 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 16:01:49.630420 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630407 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 16:01:49.630477 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630426 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 16:01:49.630477 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630433 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 16:01:49.630477 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630440 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 16:01:49.630477 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630446 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 16:01:49.630477 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.630452 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 16:01:49.631152 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.631141 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 16:01:49.631152 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.631151 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 16:01:49.631221 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.631160 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 16:01:49.631221 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.631167 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 16:01:49.631221 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.631176 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 16:01:49.631221 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.631185 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 16:01:49.632559 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.632537 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 16:01:49.632559 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.632554 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 16:01:49.636906 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.636870 2573 server.go:1263] "Started kubelet" Oct 31 16:01:49.637034 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.636962 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 16:01:49.637088 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.637010 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 16:01:49.637134 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.637091 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 16:01:49.637768 ip-10-0-142-127 systemd[1]: Started Kubernetes Kubelet. Oct 31 16:01:49.640581 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.640524 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 16:01:49.641090 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.641070 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 31 16:01:49.644200 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.644161 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 16:01:49.645907 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.645883 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 16:01:49.647197 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.647181 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 16:01:49.647354 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.647328 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" Oct 31 16:01:49.648044 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648028 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 16:01:49.648044 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648029 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 16:01:49.648044 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648059 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 16:01:49.648245 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648072 2573 factory.go:55] Registering systemd factory Oct 31 16:01:49.648245 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648092 2573 factory.go:223] Registration of the systemd container factory successfully Oct 31 16:01:49.648245 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648217 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 31 16:01:49.648245 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648226 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 31 16:01:49.648398 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648376 2573 factory.go:153] Registering CRI-O factory Oct 31 16:01:49.648398 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648389 2573 factory.go:223] Registration of the crio container factory successfully Oct 31 16:01:49.648460 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648441 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 16:01:49.648460 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648455 2573 factory.go:103] Registering Raw factory Oct 31 16:01:49.648541 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.648482 2573 manager.go:1199] Started watching for new ooms in manager Oct 31 16:01:49.649776 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.649760 2573 manager.go:319] Starting recovery of all containers Oct 31 16:01:49.661413 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.661388 2573 manager.go:324] Recovery completed Oct 31 16:01:49.665595 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.665581 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:01:49.667429 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.667397 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:01:49.667501 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.667442 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:01:49.667501 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.667454 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:01:49.668026 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.668009 2573 cpu_manager.go:226] "Starting" policy="none" Oct 31 16:01:49.668026 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.668023 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 16:01:49.668160 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.668044 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 16:01:49.668160 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.668057 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:01:49.668866 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.668850 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:01:49.668939 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.668881 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:01:49.668939 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.668898 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:01:49.669278 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.669265 2573 policy_none.go:50] "Start" Oct 31 16:01:49.669326 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.669281 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 16:01:49.669326 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.669292 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 16:01:49.670223 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.670212 2573 policy_none.go:44] "Start" Oct 31 16:01:49.715546 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.715308 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 16:01:49.726601 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.726566 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-127.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 16:01:49.726781 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.726563 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-127.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.739339 2573 manager.go:341] "Starting Device Plugin manager" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.739371 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.739383 2573 server.go:87] "Starting device plugin registration server" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.739615 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.739624 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.739752 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.739857 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.739866 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.740385 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 16:01:49.744537 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.740442 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" Oct 31 16:01:49.772576 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.771485 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-127.us-west-2.compute.internal.18e3a8a67ce25db4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-127.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-127.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:49.636836788 +0000 UTC m=+0.406790990,LastTimestamp:2026-10-31 16:01:49.636836788 +0000 UTC m=+0.406790990,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-127.us-west-2.compute.internal,}" Oct 31 16:01:49.782367 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.782335 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 16:01:49.782367 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.782335 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 16:01:49.785514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.785476 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 16:01:49.786718 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.786689 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 16:01:49.786782 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.786722 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 16:01:49.786782 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.786751 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 16:01:49.786852 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.786791 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 16:01:49.813158 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.813123 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 16:01:49.813351 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.813191 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-49995" Oct 31 16:01:49.840218 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.840188 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:01:49.841121 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.841105 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:01:49.841202 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.841141 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:01:49.841202 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.841158 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:01:49.841202 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.841186 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.886447 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.886423 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:01:49.887382 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.887362 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:01:49.888202 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.888168 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal"] Oct 31 16:01:49.888251 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.888241 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:01:49.889189 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.889172 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:01:49.889255 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.889202 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:01:49.889255 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.889215 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:01:49.890344 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.890331 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:01:49.890485 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.890469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.890518 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.890501 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:01:49.891078 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.891062 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:01:49.891141 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.891084 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:01:49.891141 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.891094 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:01:49.891141 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.891065 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:01:49.891236 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.891152 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:01:49.891236 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.891162 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:01:49.892309 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.892292 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.892385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.892318 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:01:49.893094 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.893076 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:01:49.893187 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.893100 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:01:49.893187 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.893116 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:01:49.932654 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.932617 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.936168 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.936147 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.936581 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.936566 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.936621 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:49.936590 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-127.us-west-2.compute.internal\": node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" Oct 31 16:01:49.948471 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.948441 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.949584 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.949566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.949661 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.949591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:49.949661 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:49.949610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10be822f2f376ae4d26eb7bb3d2aeb64-config\") pod \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"10be822f2f376ae4d26eb7bb3d2aeb64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:50.010685 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:50.010655 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 16:01:50.011641 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:50.011626 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.129331 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.129307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.129418 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.129339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.129418 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.129358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10be822f2f376ae4d26eb7bb3d2aeb64-config\") pod \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"10be822f2f376ae4d26eb7bb3d2aeb64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.129483 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.129423 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10be822f2f376ae4d26eb7bb3d2aeb64-config\") pod \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"10be822f2f376ae4d26eb7bb3d2aeb64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.129483 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.129449 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.129483 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.129423 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.188969 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.188934 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 16:01:51.315485 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.315436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.318396 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.318362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:01:51.615533 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.615451 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 16:01:51.615988 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.615640 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 16:01:51.615988 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:51.615707 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7918a53732334b8eb1150a335f38936-a35c44805d999c48.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-127.us-west-2.compute.internal.18e3a8a67eb572c7\": read tcp 10.0.142.127:44372->3.148.19.237:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-127.us-west-2.compute.internal.18e3a8a67eb572c7 default 4786 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-127.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-127.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-127.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 16:01:49 +0000 UTC,LastTimestamp:2026-10-31 16:01:49.889208659 +0000 UTC m=+0.659162863,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-127.us-west-2.compute.internal,}" Oct 31 16:01:51.655898 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.655860 2573 apiserver.go:52] "Watching apiserver" Oct 31 16:01:51.725732 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.725709 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 16:01:51.896519 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.896408 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 15:56:50 +0000 UTC" deadline="2028-04-08 09:31:42.639338769 +0000 UTC" Oct 31 16:01:51.896519 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.896458 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12593h29m50.742885346s" Oct 31 16:01:51.925285 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.925252 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 16:01:51.925588 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.925569 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-s7jw9","openshift-multus/multus-6wd5x","openshift-network-diagnostics/network-check-target-kddfn","openshift-ovn-kubernetes/ovnkube-node-m6xpv","kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf","openshift-image-registry/node-ca-nmnvx","openshift-multus/multus-additional-cni-plugins-hwvf5","openshift-multus/network-metrics-daemon-mmp6g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-mvztk","openshift-network-operator/iptables-alerter-9wns7","kube-system/konnectivity-agent-4wccg","openshift-cluster-node-tuning-operator/tuned-blwvk"] Oct 31 16:01:51.927155 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.927125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:51.927230 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:51.927213 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:01:51.928067 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.928051 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:51.929099 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.929075 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:51.930216 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.930200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:51.931418 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.931401 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:51.932563 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.932543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:51.933574 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.933559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:51.934591 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.934576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:51.934676 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:51.934631 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:01:51.935209 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-host\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:51.935269 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:51.935269 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9982b4-6251-4984-8493-e884e03f13ae-iptables-alerter-script\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:51.935269 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-serviceca\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:51.935360 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgj6\" (UniqueName: \"kubernetes.io/projected/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-kube-api-access-6pgj6\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:51.935388 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9982b4-6251-4984-8493-e884e03f13ae-host-slash\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:51.935419 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbvr\" (UniqueName: \"kubernetes.io/projected/eb9982b4-6251-4984-8493-e884e03f13ae-kube-api-access-9nbvr\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:51.935691 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.935676 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:51.935758 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:51.935731 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:01:51.936807 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.936782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6wd5x" Oct 31 16:01:51.937959 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.937944 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:51.939445 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.939419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:51.951855 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.951830 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:01:51.981131 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.981106 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 16:01:51.981253 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.981232 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:01:51.981344 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.981332 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 16:01:51.981443 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:51.981425 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vnjtb\"" Oct 31 16:01:52.001116 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001094 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 16:01:52.001269 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001217 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 16:01:52.001269 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001240 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msflc\"" Oct 31 16:01:52.001389 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001273 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 16:01:52.001389 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001307 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 16:01:52.001389 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001320 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 16:01:52.001389 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001335 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 16:01:52.001389 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001366 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgxcq\"" Oct 31 16:01:52.001389 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001367 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 16:01:52.001598 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001417 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 16:01:52.001598 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001533 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 16:01:52.001664 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001637 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 16:01:52.001664 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001653 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 16:01:52.001787 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001771 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d4rvf\"" Oct 31 16:01:52.001837 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001818 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 16:01:52.001837 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001832 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tcbkl\"" Oct 31 16:01:52.001906 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.001842 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 16:01:52.002963 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.002951 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 16:01:52.003069 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003056 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 16:01:52.003116 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003066 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 16:01:52.003333 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003317 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 16:01:52.003387 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003370 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9wrg2\"" Oct 31 16:01:52.003441 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003422 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnhm9\"" Oct 31 16:01:52.003441 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003425 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 16:01:52.003514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003450 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 16:01:52.003514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003474 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:01:52.003514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003493 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wg2f6\"" Oct 31 16:01:52.003514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003496 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 16:01:52.003640 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003529 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 16:01:52.003640 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.003533 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 16:01:52.024473 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.024449 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 16:01:52.024607 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.024538 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f9xg5\"" Oct 31 16:01:52.029327 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.029308 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 16:01:52.029480 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.029460 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 16:01:52.029553 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.029539 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kx9b9" Oct 31 16:01:52.036078 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-env-overrides\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.036194 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-dbus\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.036194 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-conf-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.036194 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.036194 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-lib-modules\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.036385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-host\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.036385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce582737-4c20-4d14-9199-ef952d229b5a-ovn-node-metrics-cert\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.036385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-registration-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.036385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-kubelet\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.036385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysconfig\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.036385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-multus-certs\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036416 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-var-lib-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-sys-fs\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43a99cec-f100-473d-b831-f8052ae72bf0-konnectivity-ca\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036531 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.036582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78wzd\" (UniqueName: \"kubernetes.io/projected/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-kube-api-access-78wzd\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-node-log\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-config\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-kubernetes\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-systemd\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-kubelet-config\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-k8s-cni-cncf-io\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-netns\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036834 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-tuned\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpvx\" (UniqueName: \"kubernetes.io/projected/910df55b-a9db-4aa6-acc1-3721c868beb8-kube-api-access-8tpvx\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.036901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-host\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-hostroot\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-systemd-units\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-netd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036968 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-host\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.036996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cnibin\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqs6c\" (UniqueName: \"kubernetes.io/projected/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-kube-api-access-cqs6c\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-var-lib-kubelet\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-ovn\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-bin\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-script-lib\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-socket-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cnibin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-multus\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-daemon-config\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-etc-kubernetes\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.037318 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9982b4-6251-4984-8493-e884e03f13ae-iptables-alerter-script\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-bin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a4cede-89fb-421e-be64-69f3ea494179-tmp-dir\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-netns\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-kubelet\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43a99cec-f100-473d-b831-f8052ae72bf0-agent-certs\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037534 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-serviceca\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgj6\" (UniqueName: \"kubernetes.io/projected/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-kube-api-access-6pgj6\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-systemd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037737 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrm9r\" (UniqueName: \"kubernetes.io/projected/55a6396f-e3a0-42ec-88fa-c4d23384e807-kube-api-access-wrm9r\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.037912 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-run\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-tmp\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbvr\" (UniqueName: \"kubernetes.io/projected/eb9982b4-6251-4984-8493-e884e03f13ae-kube-api-access-9nbvr\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037937 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/44c3511b-f377-4032-a196-2490d71549ae-kube-api-access-plhvw\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-system-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.037988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-conf\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-etc-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45vjp\" (UniqueName: \"kubernetes.io/projected/ce582737-4c20-4d14-9199-ef952d229b5a-kube-api-access-45vjp\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-device-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cni-binary-copy\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-socket-dir-parent\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95a4cede-89fb-421e-be64-69f3ea494179-hosts-file\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038208 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-serviceca\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9ll\" (UniqueName: \"kubernetes.io/projected/95a4cede-89fb-421e-be64-69f3ea494179-kube-api-access-5k9ll\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.038430 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-log-socket\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.038856 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9982b4-6251-4984-8493-e884e03f13ae-host-slash\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:52.038856 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038305 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-slash\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.038856 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-modprobe-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.038856 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9982b4-6251-4984-8493-e884e03f13ae-host-slash\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:52.038856 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-sys\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.038856 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.038660 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9982b4-6251-4984-8493-e884e03f13ae-iptables-alerter-script\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:52.092571 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.092546 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 16:01:52.092571 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.092571 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 16:01:52.092795 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.092583 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qbg9n for pod openshift-network-diagnostics/network-check-target-kddfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:52.092795 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.092553 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 16:01:52.092795 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.092656 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n podName:ec9fc385-70b6-442b-b076-0102e41c5fbd nodeName:}" failed. No retries permitted until 2026-10-31 16:01:52.592636116 +0000 UTC m=+2.283099681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qbg9n" (UniqueName: "kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n") pod "network-check-target-kddfn" (UID: "ec9fc385-70b6-442b-b076-0102e41c5fbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:52.095719 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.095697 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbvr\" (UniqueName: \"kubernetes.io/projected/eb9982b4-6251-4984-8493-e884e03f13ae-kube-api-access-9nbvr\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:52.095804 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.095763 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgj6\" (UniqueName: \"kubernetes.io/projected/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-kube-api-access-6pgj6\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:52.138844 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.138844 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-multus-certs\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-var-lib-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-sys-fs\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43a99cec-f100-473d-b831-f8052ae72bf0-konnectivity-ca\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138946 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138939 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-multus-certs\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-var-lib-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.138994 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-sys-fs\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wzd\" (UniqueName: \"kubernetes.io/projected/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-kube-api-access-78wzd\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.139102 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.139090 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139122 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-node-log\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-node-log\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.139175 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret podName:5a66684b-1ce2-4892-b466-d95589fcf351 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:52.639154042 +0000 UTC m=+2.329617805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret") pod "global-pull-secret-syncer-mvztk" (UID: "5a66684b-1ce2-4892-b466-d95589fcf351") : object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139186 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-config\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-kubernetes\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-systemd\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-kubelet-config\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139297 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-k8s-cni-cncf-io\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139319 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-kubernetes\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-netns\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139338 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-systemd\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-tuned\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpvx\" (UniqueName: \"kubernetes.io/projected/910df55b-a9db-4aa6-acc1-3721c868beb8-kube-api-access-8tpvx\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139393 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-k8s-cni-cncf-io\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139360 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-netns\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.139774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-hostroot\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139381 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-kubelet-config\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-systemd-units\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139463 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-systemd-units\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-netd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139500 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-netd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cnibin\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139545 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-hostroot\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs6c\" (UniqueName: \"kubernetes.io/projected/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-kube-api-access-cqs6c\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139561 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43a99cec-f100-473d-b831-f8052ae72bf0-konnectivity-ca\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-var-lib-kubelet\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-ovn\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-bin\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.139621 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139628 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-var-lib-kubelet\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cnibin\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.140596 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.139673 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:52.639658178 +0000 UTC m=+2.330121744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139674 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-ovn\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-script-lib\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139701 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-bin\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-socket-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cnibin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-multus\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139768 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-config\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-daemon-config\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-etc-kubernetes\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139825 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cnibin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139813 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-multus\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139848 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-etc-kubernetes\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139809 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-socket-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139868 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.141433 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139898 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-bin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a4cede-89fb-421e-be64-69f3ea494179-tmp-dir\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.139975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-netns\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-kubelet\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43a99cec-f100-473d-b831-f8052ae72bf0-agent-certs\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140082 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-netns\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-systemd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140119 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-systemd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140166 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140168 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-script-lib\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140179 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140002 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-bin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140204 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-kubelet\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrm9r\" (UniqueName: \"kubernetes.io/projected/55a6396f-e3a0-42ec-88fa-c4d23384e807-kube-api-access-wrm9r\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a4cede-89fb-421e-be64-69f3ea494179-tmp-dir\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-run\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-tmp\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140349 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-run\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/44c3511b-f377-4032-a196-2490d71549ae-kube-api-access-plhvw\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-system-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-conf\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-system-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140376 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-daemon-config\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-etc-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140500 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-etc-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45vjp\" (UniqueName: \"kubernetes.io/projected/ce582737-4c20-4d14-9199-ef952d229b5a-kube-api-access-45vjp\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-device-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.142872 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140558 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cni-binary-copy\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-socket-dir-parent\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140460 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140599 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95a4cede-89fb-421e-be64-69f3ea494179-hosts-file\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9ll\" (UniqueName: \"kubernetes.io/projected/95a4cede-89fb-421e-be64-69f3ea494179-kube-api-access-5k9ll\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140617 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-conf\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-log-socket\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140668 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-device-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-slash\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140693 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-socket-dir-parent\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-modprobe-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-sys\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-env-overrides\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-dbus\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-conf-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-lib-modules\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140873 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-host\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce582737-4c20-4d14-9199-ef952d229b5a-ovn-node-metrics-cert\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140904 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-registration-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-kubelet\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140979 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-conf-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysconfig\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysconfig\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141064 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-registration-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.140955 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-modprobe-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-sys\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141164 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-lib-modules\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141169 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-kubelet\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141211 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-log-socket\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141245 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-slash\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141305 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-host\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-dbus\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141420 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95a4cede-89fb-421e-be64-69f3ea494179-hosts-file\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.143850 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.141936 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cni-binary-copy\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.144332 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.142011 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-env-overrides\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.144332 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.142060 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-tuned\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.144332 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.142417 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.144332 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.142484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-tmp\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.144332 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.142837 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43a99cec-f100-473d-b831-f8052ae72bf0-agent-certs\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:52.144332 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.143657 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce582737-4c20-4d14-9199-ef952d229b5a-ovn-node-metrics-cert\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.199541 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.199455 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrm9r\" (UniqueName: \"kubernetes.io/projected/55a6396f-e3a0-42ec-88fa-c4d23384e807-kube-api-access-wrm9r\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:52.199679 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.199575 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/44c3511b-f377-4032-a196-2490d71549ae-kube-api-access-plhvw\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.199945 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.199926 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpvx\" (UniqueName: \"kubernetes.io/projected/910df55b-a9db-4aa6-acc1-3721c868beb8-kube-api-access-8tpvx\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.200021 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.199983 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wzd\" (UniqueName: \"kubernetes.io/projected/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-kube-api-access-78wzd\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.200061 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.200018 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9ll\" (UniqueName: \"kubernetes.io/projected/95a4cede-89fb-421e-be64-69f3ea494179-kube-api-access-5k9ll\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.200191 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.200175 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45vjp\" (UniqueName: \"kubernetes.io/projected/ce582737-4c20-4d14-9199-ef952d229b5a-kube-api-access-45vjp\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.201039 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.201019 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqs6c\" (UniqueName: \"kubernetes.io/projected/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-kube-api-access-cqs6c\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.261887 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.261847 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:01:52.276763 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.276735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:01:52.302373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.302341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:01:52.304098 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.304074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:01:52.333088 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.333052 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:01:52.357485 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.357456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:01:52.358551 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.358528 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6wd5x" Oct 31 16:01:52.385629 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.385596 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:52.390186 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.390165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:01:52.409154 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:52.409099 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a99cec_f100_473d_b831_f8052ae72bf0.slice/crio-79eeb9f1f1d45b72bce1e68044be92c18d92f91de780725c1058e95269a70379 WatchSource:0}: Error finding container 79eeb9f1f1d45b72bce1e68044be92c18d92f91de780725c1058e95269a70379: Status 404 returned error can't find the container with id 79eeb9f1f1d45b72bce1e68044be92c18d92f91de780725c1058e95269a70379 Oct 31 16:01:52.409544 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:52.409517 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a45f772b2761dc4954a13fe159427f7.slice/crio-05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a WatchSource:0}: Error finding container 05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a: Status 404 returned error can't find the container with id 05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a Oct 31 16:01:52.410611 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:52.410585 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910df55b_a9db_4aa6_acc1_3721c868beb8.slice/crio-6ea3c20a505035706b2346a81b24ef02721ab14894b46923f4a2d17608ded9d4 WatchSource:0}: Error finding container 6ea3c20a505035706b2346a81b24ef02721ab14894b46923f4a2d17608ded9d4: Status 404 returned error can't find the container with id 6ea3c20a505035706b2346a81b24ef02721ab14894b46923f4a2d17608ded9d4 Oct 31 16:01:52.411279 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:52.411259 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10be822f2f376ae4d26eb7bb3d2aeb64.slice/crio-9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00 WatchSource:0}: Error finding container 9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00: Status 404 returned error can't find the container with id 9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00 Oct 31 16:01:52.414029 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.413972 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:01:52.463548 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.463472 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:01:52.488319 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.488288 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:01:52.643884 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.643845 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.643912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.643944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644019 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644042 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644055 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qbg9n for pod openshift-network-diagnostics/network-check-target-kddfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644081 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644106 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n podName:ec9fc385-70b6-442b-b076-0102e41c5fbd nodeName:}" failed. No retries permitted until 2026-10-31 16:01:53.644093155 +0000 UTC m=+3.334556703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbg9n" (UniqueName: "kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n") pod "network-check-target-kddfn" (UID: "ec9fc385-70b6-442b-b076-0102e41c5fbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644116 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644158 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret podName:5a66684b-1ce2-4892-b466-d95589fcf351 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:53.644124762 +0000 UTC m=+3.334588325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret") pod "global-pull-secret-syncer-mvztk" (UID: "5a66684b-1ce2-4892-b466-d95589fcf351") : object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:52.644539 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:52.644175 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:53.644167733 +0000 UTC m=+3.334631282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:52.870029 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.869937 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wccg" event={"ID":"43a99cec-f100-473d-b831-f8052ae72bf0","Type":"ContainerStarted","Data":"79eeb9f1f1d45b72bce1e68044be92c18d92f91de780725c1058e95269a70379"} Oct 31 16:01:52.871316 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.871291 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"3a45f772b2761dc4954a13fe159427f7","Type":"ContainerStarted","Data":"05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a"} Oct 31 16:01:52.873577 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.873545 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"10be822f2f376ae4d26eb7bb3d2aeb64","Type":"ContainerStarted","Data":"9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00"} Oct 31 16:01:52.875019 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:52.874997 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-blwvk" event={"ID":"910df55b-a9db-4aa6-acc1-3721c868beb8","Type":"ContainerStarted","Data":"6ea3c20a505035706b2346a81b24ef02721ab14894b46923f4a2d17608ded9d4"} Oct 31 16:01:53.031702 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.031622 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:51 +0000 UTC" deadline="2028-07-31 14:27:10.041153328 +0000 UTC" Oct 31 16:01:53.031702 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.031696 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15334h25m17.009461456s" Oct 31 16:01:53.171201 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:53.171169 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9982b4_6251_4984_8493_e884e03f13ae.slice/crio-ea6406e6f83cc72bc1721251e525217ab0c964d4145bc634e18f96dce94e5cac WatchSource:0}: Error finding container ea6406e6f83cc72bc1721251e525217ab0c964d4145bc634e18f96dce94e5cac: Status 404 returned error can't find the container with id ea6406e6f83cc72bc1721251e525217ab0c964d4145bc634e18f96dce94e5cac Oct 31 16:01:53.175257 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:53.175230 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb6cc68_a2af_438f_83be_9f1e9afc57b6.slice/crio-72d608b8c51f0f4b7739bcbbfb265a18ff4e44729982f6dd8d718e81f3941986 WatchSource:0}: Error finding container 72d608b8c51f0f4b7739bcbbfb265a18ff4e44729982f6dd8d718e81f3941986: Status 404 returned error can't find the container with id 72d608b8c51f0f4b7739bcbbfb265a18ff4e44729982f6dd8d718e81f3941986 Oct 31 16:01:53.175828 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:53.175802 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c3511b_f377_4032_a196_2490d71549ae.slice/crio-c7fb3ca298b02979ab13cf08f753b95ba974fcc728ed61d74042de4de444e1b0 WatchSource:0}: Error finding container c7fb3ca298b02979ab13cf08f753b95ba974fcc728ed61d74042de4de444e1b0: Status 404 returned error can't find the container with id c7fb3ca298b02979ab13cf08f753b95ba974fcc728ed61d74042de4de444e1b0 Oct 31 16:01:53.219809 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:53.219778 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce582737_4c20_4d14_9199_ef952d229b5a.slice/crio-538c4450eb30426950288b75c8b5a200cbf34315f2068917609e0fa204cb08ff WatchSource:0}: Error finding container 538c4450eb30426950288b75c8b5a200cbf34315f2068917609e0fa204cb08ff: Status 404 returned error can't find the container with id 538c4450eb30426950288b75c8b5a200cbf34315f2068917609e0fa204cb08ff Oct 31 16:01:53.249348 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:53.249322 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a4cede_89fb_421e_be64_69f3ea494179.slice/crio-0c398d0815dd3229bb73cf811951de1b35f48e67093b9f0b33f46cf305d20313 WatchSource:0}: Error finding container 0c398d0815dd3229bb73cf811951de1b35f48e67093b9f0b33f46cf305d20313: Status 404 returned error can't find the container with id 0c398d0815dd3229bb73cf811951de1b35f48e67093b9f0b33f46cf305d20313 Oct 31 16:01:53.257959 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:01:53.257936 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6483e8c0_5b7a_46c6_b270_1f741dd0e8ad.slice/crio-6c92a3bb8e43c0f1642447fdd8761494e77e3363a51003629f23d30add3a8722 WatchSource:0}: Error finding container 6c92a3bb8e43c0f1642447fdd8761494e77e3363a51003629f23d30add3a8722: Status 404 returned error can't find the container with id 6c92a3bb8e43c0f1642447fdd8761494e77e3363a51003629f23d30add3a8722 Oct 31 16:01:53.650111 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.650071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:53.650784 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.650158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:53.650784 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.650221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:53.650784 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650346 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:53.650784 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650412 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret podName:5a66684b-1ce2-4892-b466-d95589fcf351 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:55.650392461 +0000 UTC m=+5.340856010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret") pod "global-pull-secret-syncer-mvztk" (UID: "5a66684b-1ce2-4892-b466-d95589fcf351") : object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:53.650784 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650766 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:53.651095 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650818 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:55.650803042 +0000 UTC m=+5.341266593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:53.651095 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650902 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 16:01:53.651095 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650918 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 16:01:53.651095 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650930 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qbg9n for pod openshift-network-diagnostics/network-check-target-kddfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:53.651095 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.650969 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n podName:ec9fc385-70b6-442b-b076-0102e41c5fbd nodeName:}" failed. No retries permitted until 2026-10-31 16:01:55.65095621 +0000 UTC m=+5.341419763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbg9n" (UniqueName: "kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n") pod "network-check-target-kddfn" (UID: "ec9fc385-70b6-442b-b076-0102e41c5fbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:53.867485 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.867442 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:53.867674 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.867574 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:01:53.867674 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.867658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:53.867791 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.867725 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:01:53.867791 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.867781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:53.867886 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:53.867857 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:01:53.878715 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.878674 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7jw9" event={"ID":"95a4cede-89fb-421e-be64-69f3ea494179","Type":"ContainerStarted","Data":"0c398d0815dd3229bb73cf811951de1b35f48e67093b9f0b33f46cf305d20313"} Oct 31 16:01:53.881807 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.881778 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerStarted","Data":"ba3ec2f618b10d953c518daac66851775f06f8a32bf325dd2b61dda04195ac2c"} Oct 31 16:01:53.883437 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.883412 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nmnvx" event={"ID":"9cb6cc68-a2af-438f-83be-9f1e9afc57b6","Type":"ContainerStarted","Data":"72d608b8c51f0f4b7739bcbbfb265a18ff4e44729982f6dd8d718e81f3941986"} Oct 31 16:01:53.892299 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.892259 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6wd5x" event={"ID":"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad","Type":"ContainerStarted","Data":"6c92a3bb8e43c0f1642447fdd8761494e77e3363a51003629f23d30add3a8722"} Oct 31 16:01:53.894317 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.894290 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"538c4450eb30426950288b75c8b5a200cbf34315f2068917609e0fa204cb08ff"} Oct 31 16:01:53.895856 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.895832 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" event={"ID":"44c3511b-f377-4032-a196-2490d71549ae","Type":"ContainerStarted","Data":"c7fb3ca298b02979ab13cf08f753b95ba974fcc728ed61d74042de4de444e1b0"} Oct 31 16:01:53.897903 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:53.897880 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wns7" event={"ID":"eb9982b4-6251-4984-8493-e884e03f13ae","Type":"ContainerStarted","Data":"ea6406e6f83cc72bc1721251e525217ab0c964d4145bc634e18f96dce94e5cac"} Oct 31 16:01:54.032617 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:54.032507 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:51 +0000 UTC" deadline="2028-08-07 08:29:09.915061724 +0000 UTC" Oct 31 16:01:54.032617 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:54.032571 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15496h27m15.882495148s" Oct 31 16:01:55.664444 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:55.664405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:55.664495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:55.664553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664673 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664738 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret podName:5a66684b-1ce2-4892-b466-d95589fcf351 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:59.664720809 +0000 UTC m=+9.355184364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret") pod "global-pull-secret-syncer-mvztk" (UID: "5a66684b-1ce2-4892-b466-d95589fcf351") : object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664806 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664838 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:01:59.664827276 +0000 UTC m=+9.355290828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664914 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664927 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 16:01:55.664943 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664940 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qbg9n for pod openshift-network-diagnostics/network-check-target-kddfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:55.665515 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.664983 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n podName:ec9fc385-70b6-442b-b076-0102e41c5fbd nodeName:}" failed. No retries permitted until 2026-10-31 16:01:59.66497263 +0000 UTC m=+9.355436194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbg9n" (UniqueName: "kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n") pod "network-check-target-kddfn" (UID: "ec9fc385-70b6-442b-b076-0102e41c5fbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:55.867246 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:55.867216 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:55.867493 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.867359 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:01:55.867493 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:55.867419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:55.867493 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.867488 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:01:55.867695 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:55.867526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:55.867695 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:55.867590 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:01:56.928682 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:56.928447 2573 generic.go:356] "Generic (PLEG): container finished" podID="3a45f772b2761dc4954a13fe159427f7" containerID="a35766bb434ccc7851b7bddb23d31e53cab4abf2e98d5e786e14f1c3191c3978" exitCode=0 Oct 31 16:01:56.929242 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:56.929209 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"3a45f772b2761dc4954a13fe159427f7","Type":"ContainerDied","Data":"a35766bb434ccc7851b7bddb23d31e53cab4abf2e98d5e786e14f1c3191c3978"} Oct 31 16:01:56.939055 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:56.939021 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerStarted","Data":"0202b4be9b346f2a69a3ecc7aa86ffc678ff9c071bc05a5ef056eff6390ee131"} Oct 31 16:01:57.867070 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.867033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:57.867272 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:57.867182 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:01:57.867272 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.867263 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:57.867402 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:57.867332 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:01:57.867402 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.867387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:57.867499 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:57.867465 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:01:57.947469 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.947359 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"10be822f2f376ae4d26eb7bb3d2aeb64","Type":"ContainerStarted","Data":"0a54cf93dfe48cc6e8eefc6b6d77027f317d5800525253d4ebd7ec45d7c9f0cc"} Oct 31 16:01:57.957741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.957696 2573 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="0202b4be9b346f2a69a3ecc7aa86ffc678ff9c071bc05a5ef056eff6390ee131" exitCode=0 Oct 31 16:01:57.957897 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.957780 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerDied","Data":"0202b4be9b346f2a69a3ecc7aa86ffc678ff9c071bc05a5ef056eff6390ee131"} Oct 31 16:01:57.961437 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.961412 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4wccg" event={"ID":"43a99cec-f100-473d-b831-f8052ae72bf0","Type":"ContainerStarted","Data":"a34f94e4e253e134a2df485495f3a0a64f96d59f8a381192b46dd7e5fd95e747"} Oct 31 16:01:57.967991 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:57.967965 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"3a45f772b2761dc4954a13fe159427f7","Type":"ContainerStarted","Data":"8634b55a03d4b691ac186ca506faac98236b737b8be67ed3c93e87c2645625eb"} Oct 31 16:01:58.170023 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:58.169976 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:58.194078 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:58.194007 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" podStartSLOduration=7.193988437 podStartE2EDuration="7.193988437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:01:58.076621278 +0000 UTC m=+7.767084841" watchObservedRunningTime="2026-10-31 16:01:58.193988437 +0000 UTC m=+7.884452010" Oct 31 16:01:58.420219 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:58.419341 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4wccg" podStartSLOduration=3.028076304 podStartE2EDuration="7.419317521s" totalImagesPullingTime="4.391241217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:01:58.307317921 +0000 UTC m=+7.997781495" watchObservedRunningTime="2026-10-31 16:01:58.419317521 +0000 UTC m=+8.109781093" Oct 31 16:01:58.865733 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:58.865617 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:58.866119 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:58.866098 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:58.979463 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:58.979414 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" podStartSLOduration=7.97939334 podStartE2EDuration="7.97939334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:01:58.420536945 +0000 UTC m=+8.111000519" watchObservedRunningTime="2026-10-31 16:01:58.97939334 +0000 UTC m=+8.669856910" Oct 31 16:01:58.984514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:58.984329 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:01:59.701386 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.701357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:59.701591 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.701398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:59.701591 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.701424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:59.701591 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701542 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 16:01:59.701591 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701557 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 16:01:59.701591 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701569 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qbg9n for pod openshift-network-diagnostics/network-check-target-kddfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:59.701896 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701623 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n podName:ec9fc385-70b6-442b-b076-0102e41c5fbd nodeName:}" failed. No retries permitted until 2026-10-31 16:02:07.701606563 +0000 UTC m=+17.392070112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbg9n" (UniqueName: "kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n") pod "network-check-target-kddfn" (UID: "ec9fc385-70b6-442b-b076-0102e41c5fbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:01:59.701896 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701636 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:59.701896 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701701 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret podName:5a66684b-1ce2-4892-b466-d95589fcf351 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:07.701682723 +0000 UTC m=+17.392146274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret") pod "global-pull-secret-syncer-mvztk" (UID: "5a66684b-1ce2-4892-b466-d95589fcf351") : object "kube-system"/"original-pull-secret" not registered Oct 31 16:01:59.701896 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701770 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:59.701896 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.701803 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:07.701792146 +0000 UTC m=+17.392255700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:01:59.867131 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.867056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:01:59.867311 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.867197 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:01:59.867311 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.867277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:01:59.867434 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.867347 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:01:59.867434 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.867424 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:01:59.867565 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:01:59.867542 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:01:59.977107 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.977061 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nmnvx" event={"ID":"9cb6cc68-a2af-438f-83be-9f1e9afc57b6","Type":"ContainerStarted","Data":"7212110076fdf11dc436002885269a7a4cf8ea3b313e3a9108a4721e5e5c39a1"} Oct 31 16:01:59.982798 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:01:59.982753 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" event={"ID":"44c3511b-f377-4032-a196-2490d71549ae","Type":"ContainerStarted","Data":"a22c1247056b9e293a72d81077e7718a1ed540029c328bee393d9de6b7596d0a"} Oct 31 16:02:00.092041 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:00.091979 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nmnvx" podStartSLOduration=3.474883163 podStartE2EDuration="9.091960305s" totalImagesPullingTime="5.617077142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:00.091514021 +0000 UTC m=+9.781977598" watchObservedRunningTime="2026-10-31 16:02:00.091960305 +0000 UTC m=+9.782423878" Oct 31 16:02:01.867438 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:01.867116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:01.867438 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:01.867171 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:01.867438 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:01.867262 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:01.867438 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:01.867309 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:01.867438 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:01.867318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:01.867438 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:01.867406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:01.942061 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:01.942031 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 16:02:01.991211 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:01.990131 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7jw9" event={"ID":"95a4cede-89fb-421e-be64-69f3ea494179","Type":"ContainerStarted","Data":"9bb31283c24dbaef903dbc294ef10939e923561fdc158fccc560e1c4a511b51b"} Oct 31 16:02:01.992689 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:01.992660 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6wd5x" event={"ID":"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad","Type":"ContainerStarted","Data":"4e1ca33c474ba876b0fb84795f631338177c16d80eea13ba8a22a16f459db12c"} Oct 31 16:02:01.995240 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:01.995217 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" event={"ID":"44c3511b-f377-4032-a196-2490d71549ae","Type":"ContainerStarted","Data":"894c5b70b225b34037942664d1a2c02978ce878ce9c3efcc24991c33bd94b146"} Oct 31 16:02:02.102288 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:02.102234 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s7jw9" podStartSLOduration=2.645930923 podStartE2EDuration="11.102220151s" totalImagesPullingTime="8.456289228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:02.102168051 +0000 UTC m=+11.792631620" watchObservedRunningTime="2026-10-31 16:02:02.102220151 +0000 UTC m=+11.792683721" Oct 31 16:02:02.216124 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:02.216063 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6wd5x" podStartSLOduration=3.195817806 podStartE2EDuration="11.216043217s" totalImagesPullingTime="8.020225411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:02.215792983 +0000 UTC m=+11.906256580" watchObservedRunningTime="2026-10-31 16:02:02.216043217 +0000 UTC m=+11.906506793" Oct 31 16:02:02.827102 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:02.826748 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T16:02:01.942058248Z","UUID":"caebfd67-45d3-427b-a162-669b607a4129","Handler":null,"Name":"","Endpoint":""} Oct 31 16:02:02.828343 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:02.828322 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 16:02:02.828452 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:02.828351 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 16:02:02.999901 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:02.999869 2573 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="7664a9a552a773662349992f61d1f71deb09fffd34dfd35f381ec1264a19440b" exitCode=0 Oct 31 16:02:03.000334 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:02.999949 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerDied","Data":"7664a9a552a773662349992f61d1f71deb09fffd34dfd35f381ec1264a19440b"} Oct 31 16:02:03.866738 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:03.866655 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:03.866738 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:03.866685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:03.866954 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:03.866775 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:03.867108 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:03.867078 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:03.867257 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:03.867194 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:03.867324 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:03.867282 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:04.004733 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:04.004696 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" event={"ID":"44c3511b-f377-4032-a196-2490d71549ae","Type":"ContainerStarted","Data":"9a7a18ab086b01ed300d9b1565f0be003cc860b4611dcfe350d2f5d7143279ed"} Oct 31 16:02:04.007175 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:04.007148 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wns7" event={"ID":"eb9982b4-6251-4984-8493-e884e03f13ae","Type":"ContainerStarted","Data":"476a69773f9834645ec578f51448f62b6ee14baefa9faf8636c86fa1e7d31aaa"} Oct 31 16:02:04.008674 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:04.008652 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-blwvk" event={"ID":"910df55b-a9db-4aa6-acc1-3721c868beb8","Type":"ContainerStarted","Data":"b81ce88e07e51fc8fff970c02547c806b5d427879d28b471ca2e1188ad3a77e1"} Oct 31 16:02:04.119844 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:04.119751 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" podStartSLOduration=3.098031247 podStartE2EDuration="13.119737131s" totalImagesPullingTime="10.021705884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:04.119215184 +0000 UTC m=+13.809678756" watchObservedRunningTime="2026-10-31 16:02:04.119737131 +0000 UTC m=+13.810200836" Oct 31 16:02:04.232865 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:04.232811 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9wns7" podStartSLOduration=5.551071786 podStartE2EDuration="14.232799255s" totalImagesPullingTime="8.681727469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:04.232403825 +0000 UTC m=+13.922867422" watchObservedRunningTime="2026-10-31 16:02:04.232799255 +0000 UTC m=+13.923262826" Oct 31 16:02:04.346025 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:04.345979 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-blwvk" podStartSLOduration=2.784149047 podStartE2EDuration="13.345966714s" totalImagesPullingTime="10.561817667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:04.345766339 +0000 UTC m=+14.036229916" watchObservedRunningTime="2026-10-31 16:02:04.345966714 +0000 UTC m=+14.036430284" Oct 31 16:02:05.013752 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:05.013717 2573 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="2497e714c76f977665ec9b4b4cab0d7bf8b540710d962b43da5b5c7a9dc14c07" exitCode=0 Oct 31 16:02:05.014352 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:05.014065 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerDied","Data":"2497e714c76f977665ec9b4b4cab0d7bf8b540710d962b43da5b5c7a9dc14c07"} Oct 31 16:02:05.867017 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:05.866985 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:05.867204 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:05.867099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:05.867204 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:05.867108 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:05.867297 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:05.867209 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:05.867297 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:05.867104 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:05.867379 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:05.867303 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:07.021362 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.021061 2573 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="5840407302e9d1e0b1b647c250bd99a99a812ff4277fe6b20f36dde6371a7c80" exitCode=0 Oct 31 16:02:07.021362 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.021103 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerDied","Data":"5840407302e9d1e0b1b647c250bd99a99a812ff4277fe6b20f36dde6371a7c80"} Oct 31 16:02:07.766784 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.766749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:07.766784 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.766791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:07.767022 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.766816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:07.767022 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.766917 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 16:02:07.767022 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.766987 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret podName:5a66684b-1ce2-4892-b466-d95589fcf351 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.766966186 +0000 UTC m=+33.457429753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret") pod "global-pull-secret-syncer-mvztk" (UID: "5a66684b-1ce2-4892-b466-d95589fcf351") : object "kube-system"/"original-pull-secret" not registered Oct 31 16:02:07.767022 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.766996 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:02:07.767238 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.767042 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.767030882 +0000 UTC m=+33.457494431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 16:02:07.767238 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.766928 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 16:02:07.767238 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.767062 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 16:02:07.767238 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.767071 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qbg9n for pod openshift-network-diagnostics/network-check-target-kddfn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:02:07.767238 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.767094 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n podName:ec9fc385-70b6-442b-b076-0102e41c5fbd nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.767086203 +0000 UTC m=+33.457549751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbg9n" (UniqueName: "kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n") pod "network-check-target-kddfn" (UID: "ec9fc385-70b6-442b-b076-0102e41c5fbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 16:02:07.866614 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.866574 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:07.866759 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.866586 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:07.866759 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.866691 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:07.866861 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.866759 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:07.866861 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:07.866593 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:07.866948 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:07.866859 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:09.028131 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.027957 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"77c51bde2e11cd4d64cbd2dce4f864c586a146291d4a56c26730266814ead187"} Oct 31 16:02:09.028543 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.028214 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"6b3eb6dac89201acf46e0f5c74af23c72027f55a913aa14b173ba2ad92314939"} Oct 31 16:02:09.028543 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.028251 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"5f36982f601fd87841cd0f6662076a6b5a69a7c99899dabf18064962c41107a9"} Oct 31 16:02:09.028543 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.028279 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"7b697e4bd2d2936821c988a0430e6578b62da302087cae1417449ab4a30eabe7"} Oct 31 16:02:09.028543 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.028311 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"9a34289489c68c6b067d66e2158fff0908ff80b8ddcc66b9315157a77da9210b"} Oct 31 16:02:09.028543 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.028339 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"55e2aa82d52c77ba94121304344fa38520d2f77124122369a8b12a50e1971c06"} Oct 31 16:02:09.867239 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.867155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:09.867239 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.867182 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:09.867239 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:09.867193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:09.867499 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:09.867274 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:09.867499 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:09.867368 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:09.867499 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:09.867450 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:10.032419 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:10.032390 2573 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="3262a0f36fb2867ed33c589c28a1862b244992ded17c11ce8df1fef9ebf61e0e" exitCode=0 Oct 31 16:02:10.032876 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:10.032442 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerDied","Data":"3262a0f36fb2867ed33c589c28a1862b244992ded17c11ce8df1fef9ebf61e0e"} Oct 31 16:02:11.041439 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:11.041406 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"aa07af4750a8576b273cc47248c450e024d39b61a1a96e422db34ea841d6b969"} Oct 31 16:02:11.044478 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:11.044451 2573 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="8ab2e984d27532ccecdd52bf8c66804c13a01678ad320e5d7d517bebb63c1413" exitCode=0 Oct 31 16:02:11.044603 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:11.044485 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerDied","Data":"8ab2e984d27532ccecdd52bf8c66804c13a01678ad320e5d7d517bebb63c1413"} Oct 31 16:02:11.866557 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:11.866525 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:11.866697 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:11.866623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:11.866697 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:11.866629 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:11.866781 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:11.866705 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:11.866781 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:11.866710 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:11.866781 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:11.866772 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:12.050147 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:12.050100 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" event={"ID":"37a8427f-f59a-4c8e-8c1a-67791b68f4a5","Type":"ContainerStarted","Data":"006cc2bdf15e01e9d5615c2f16fa34b995cdcab24575f6e31442430b5fd33172"} Oct 31 16:02:12.166926 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:12.166879 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hwvf5" podStartSLOduration=7.593968936 podStartE2EDuration="21.166865698s" totalImagesPullingTime="13.572896762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:12.166787824 +0000 UTC m=+21.857251397" watchObservedRunningTime="2026-10-31 16:02:12.166865698 +0000 UTC m=+21.857329268" Oct 31 16:02:13.867260 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:13.867212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:13.867260 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:13.867259 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:13.867779 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:13.867213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:13.867779 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:13.867335 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:13.867779 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:13.867385 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:13.867779 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:13.867465 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:14.057917 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:14.057887 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" event={"ID":"ce582737-4c20-4d14-9199-ef952d229b5a","Type":"ContainerStarted","Data":"ce16336a3acc6b1af4f978e275c5bfb688ffbe557c09b0a2bea4516c3018fc08"} Oct 31 16:02:14.058187 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:14.058116 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:14.073431 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:14.073406 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:14.335730 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:14.335681 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" podStartSLOduration=8.007150994 podStartE2EDuration="23.335666538s" totalImagesPullingTime="15.328515544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:14.335268817 +0000 UTC m=+24.025732390" watchObservedRunningTime="2026-10-31 16:02:14.335666538 +0000 UTC m=+24.026130109" Oct 31 16:02:15.060602 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.060558 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 16:02:15.061025 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.061008 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:15.075674 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.075648 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:15.627554 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.627522 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvztk"] Oct 31 16:02:15.627753 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.627632 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:15.627753 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:15.627714 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:15.630854 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.630836 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kddfn"] Oct 31 16:02:15.630950 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.630908 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:15.630984 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:15.630970 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:15.631347 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.631330 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mmp6g"] Oct 31 16:02:15.631428 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:15.631417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:15.631508 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:15.631492 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:16.062897 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:16.062865 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 16:02:16.866845 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:16.866810 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:16.867013 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:16.866970 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:16.867060 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:16.867015 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:16.867118 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:16.867103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:17.070263 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:17.070232 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 16:02:17.867524 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:17.867491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:17.867681 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:17.867616 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:18.866968 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:18.866933 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:18.867573 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:18.866981 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:18.867573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:18.867073 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:18.867573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:18.867198 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:19.867377 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:19.867336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:19.867754 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:19.867455 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:20.867385 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:20.867357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:20.867896 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:20.867433 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:20.867896 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:20.867492 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mvztk" podUID="5a66684b-1ce2-4892-b466-d95589fcf351" Oct 31 16:02:20.867896 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:20.867491 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mmp6g" podUID="55a6396f-e3a0-42ec-88fa-c4d23384e807" Oct 31 16:02:20.941830 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:20.941793 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:20.942054 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:20.942039 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 16:02:20.952597 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:20.952574 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:21.866868 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:21.866835 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:21.867063 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:21.866957 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kddfn" podUID="ec9fc385-70b6-442b-b076-0102e41c5fbd" Oct 31 16:02:22.074345 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.074316 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeReady" Oct 31 16:02:22.074722 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.074425 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 16:02:22.154829 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.154799 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9cdsm"] Oct 31 16:02:22.158216 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.158194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" Oct 31 16:02:22.158334 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.158240 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x72jc"] Oct 31 16:02:22.160942 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.160924 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69955"] Oct 31 16:02:22.161056 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.161041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x72jc" Oct 31 16:02:22.163675 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.163655 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-69955" Oct 31 16:02:22.186788 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.186762 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-746fc4dd59-qfxr9"] Oct 31 16:02:22.189493 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.189472 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-bthm6"] Oct 31 16:02:22.189611 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.189591 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.193973 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.193946 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b9f8d5c7-c4czr"] Oct 31 16:02:22.194254 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.194238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.197458 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.197436 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m76mb"] Oct 31 16:02:22.197576 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.197560 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.200243 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.200224 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9zgx8"] Oct 31 16:02:22.200359 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.200341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:22.202940 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.202920 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8"] Oct 31 16:02:22.203034 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.202970 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.205707 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.205685 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl"] Oct 31 16:02:22.205865 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.205843 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" Oct 31 16:02:22.208565 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.208543 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs"] Oct 31 16:02:22.208668 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.208635 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.210582 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.210563 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.210667 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.210584 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.211376 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.211361 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kns59"] Oct 31 16:02:22.211501 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.211483 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.214038 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.214023 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw"] Oct 31 16:02:22.214163 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.214148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.217703 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.217685 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-98675d9cb-646m2"] Oct 31 16:02:22.217921 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.217809 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.220459 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.220443 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq"] Oct 31 16:02:22.220578 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.220564 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.224123 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.224107 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sgkl6"] Oct 31 16:02:22.224237 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.224212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" Oct 31 16:02:22.227997 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.227979 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b2gc9"] Oct 31 16:02:22.228123 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.228108 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.231825 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.231804 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9cdsm"] Oct 31 16:02:22.231943 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.231876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.239416 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.239395 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69955"] Oct 31 16:02:22.242731 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.242705 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x72jc"] Oct 31 16:02:22.243041 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.243025 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 16:02:22.243168 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.243156 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 16:02:22.243222 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.243200 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 16:02:22.249637 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.249596 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nfvld\"" Oct 31 16:02:22.250773 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.250751 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-bthm6"] Oct 31 16:02:22.254303 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.254280 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-746fc4dd59-qfxr9"] Oct 31 16:02:22.256547 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.256524 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9zgx8"] Oct 31 16:02:22.262648 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhk2\" (UniqueName: \"kubernetes.io/projected/fb4e8bad-6ee9-4270-a117-70210189abe1-kube-api-access-xhhk2\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.262752 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e3677f-537d-4cd9-8b72-3486e64246b6-serving-cert\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.262752 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.262752 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.262867 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:22.262867 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.262867 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-snapshots\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.263014 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.263014 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.262970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vm2\" (UniqueName: \"kubernetes.io/projected/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-kube-api-access-29vm2\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.263096 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.263096 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb4e8bad-6ee9-4270-a117-70210189abe1-serving-cert\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.263182 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb926\" (UniqueName: \"kubernetes.io/projected/be3d02e8-cb60-4762-b79f-e29afc5b53ac-kube-api-access-qb926\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.263182 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.263251 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.263251 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263208 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-default-certificate\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.263251 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96z7w\" (UniqueName: \"kubernetes.io/projected/efec2af7-bad8-43d3-8fd8-45071136bdcf-kube-api-access-96z7w\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.263348 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.263379 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9cad0b0-2b17-48f6-9c14-12a419791b1f-serving-cert\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.263410 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.263441 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e3677f-537d-4cd9-8b72-3486e64246b6-config\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.263441 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgn85\" (UniqueName: \"kubernetes.io/projected/ed820d3a-f16c-4bf0-a30e-85ad2732ab58-kube-api-access-rgn85\") pod \"kas-connection-checker-685895c7c9-9cdsm\" (UID: \"ed820d3a-f16c-4bf0-a30e-85ad2732ab58\") " pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" Oct 31 16:02:22.263514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.263514 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86bf6\" (UniqueName: \"kubernetes.io/projected/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222-kube-api-access-86bf6\") pod \"volume-data-source-validator-58c9f5f46d-xx5j8\" (UID: \"7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" Oct 31 16:02:22.263592 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263520 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.263592 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.263671 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.263671 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-config\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.263671 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpgjb\" (UniqueName: \"kubernetes.io/projected/45d11ee0-d36c-4f4a-915c-fcedef344e68-kube-api-access-fpgjb\") pod \"kas-connection-checker-685895c7c9-69955\" (UID: \"45d11ee0-d36c-4f4a-915c-fcedef344e68\") " pod="kube-system/kas-connection-checker-685895c7c9-69955" Oct 31 16:02:22.263770 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-service-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.263806 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.263806 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hm7f\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.263879 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db4c92c3-fe37-4fbf-be54-a0196309c59c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:22.263879 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-tmp\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.263879 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263839 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468a8f95-953e-403c-a2d7-931d530da8ee-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.263992 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.263992 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6st7s\" (UniqueName: \"kubernetes.io/projected/b3d75aa1-14ee-4dd6-8216-440abcd707fa-kube-api-access-6st7s\") pod \"kas-connection-checker-685895c7c9-x72jc\" (UID: \"b3d75aa1-14ee-4dd6-8216-440abcd707fa\") " pod="kube-system/kas-connection-checker-685895c7c9-x72jc" Oct 31 16:02:22.263992 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.263972 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5p2\" (UniqueName: \"kubernetes.io/projected/468a8f95-953e-403c-a2d7-931d530da8ee-kube-api-access-2g5p2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.264096 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468a8f95-953e-403c-a2d7-931d530da8ee-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.264096 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzzb\" (UniqueName: \"kubernetes.io/projected/e9cad0b0-2b17-48f6-9c14-12a419791b1f-kube-api-access-bdzzb\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.264184 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.264184 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.264184 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.264303 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-stats-auth\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.264303 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wsd9\" (UniqueName: \"kubernetes.io/projected/44e3677f-537d-4cd9-8b72-3486e64246b6-kube-api-access-4wsd9\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.264303 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.264426 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5g9\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.264426 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264364 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.264426 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264395 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.264560 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.264434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.286633 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.286603 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs"] Oct 31 16:02:22.290238 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.290210 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl"] Oct 31 16:02:22.290602 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.290580 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kns59"] Oct 31 16:02:22.291274 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.291256 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-98675d9cb-646m2"] Oct 31 16:02:22.292500 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.292476 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b9f8d5c7-c4czr"] Oct 31 16:02:22.300293 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.300271 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq"] Oct 31 16:02:22.303526 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.303505 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8"] Oct 31 16:02:22.311654 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311624 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 16:02:22.311654 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311644 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.311857 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311681 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.311857 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311717 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 16:02:22.311857 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311744 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 16:02:22.311857 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311747 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.311857 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311793 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.312014 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.311957 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 16:02:22.312055 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312026 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.312055 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312033 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.312165 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312130 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312263 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312232 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312398 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312376 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.312398 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312388 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 16:02:22.312398 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312388 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 16:02:22.312551 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312532 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.312551 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312544 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.312632 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312553 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312632 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312553 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312632 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312576 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312632 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312592 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312632 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312596 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312632 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312604 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.312801 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312637 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312801 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312642 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:22.312801 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.312649 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 16:02:22.335849 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.335823 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 16:02:22.338792 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.338771 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:22.362899 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.362869 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 16:02:22.364868 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.364848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.364982 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.364878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86bf6\" (UniqueName: \"kubernetes.io/projected/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222-kube-api-access-86bf6\") pod \"volume-data-source-validator-58c9f5f46d-xx5j8\" (UID: \"7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" Oct 31 16:02:22.364982 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.364904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.364982 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.364921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.364982 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.364939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-tmp-dir\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.364982 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.364970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.365236 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.364990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-config\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.365236 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgjb\" (UniqueName: \"kubernetes.io/projected/45d11ee0-d36c-4f4a-915c-fcedef344e68-kube-api-access-fpgjb\") pod \"kas-connection-checker-685895c7c9-69955\" (UID: \"45d11ee0-d36c-4f4a-915c-fcedef344e68\") " pod="kube-system/kas-connection-checker-685895c7c9-69955" Oct 31 16:02:22.365236 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqp5\" (UniqueName: \"kubernetes.io/projected/f5e06bd6-7368-4e48-8732-782df89e89dd-kube-api-access-5mqp5\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.365236 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-service-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.365236 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365223 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sd5\" (UniqueName: \"kubernetes.io/projected/65899ffa-6ce0-4db1-8759-b17f99943f51-kube-api-access-j9sd5\") pod \"network-check-source-6f78d59f99-hlsdq\" (UID: \"65899ffa-6ce0-4db1-8759-b17f99943f51\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" Oct 31 16:02:22.365407 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365242 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8vd\" (UniqueName: \"kubernetes.io/projected/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-kube-api-access-mk8vd\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.365407 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.365474 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hm7f\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.365474 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db4c92c3-fe37-4fbf-be54-a0196309c59c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:22.365563 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-tmp\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.365563 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468a8f95-953e-403c-a2d7-931d530da8ee-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.365563 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.365707 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6st7s\" (UniqueName: \"kubernetes.io/projected/b3d75aa1-14ee-4dd6-8216-440abcd707fa-kube-api-access-6st7s\") pod \"kas-connection-checker-685895c7c9-x72jc\" (UID: \"b3d75aa1-14ee-4dd6-8216-440abcd707fa\") " pod="kube-system/kas-connection-checker-685895c7c9-x72jc" Oct 31 16:02:22.365707 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5p2\" (UniqueName: \"kubernetes.io/projected/468a8f95-953e-403c-a2d7-931d530da8ee-kube-api-access-2g5p2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.365707 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468a8f95-953e-403c-a2d7-931d530da8ee-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.365707 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365659 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.365707 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e06bd6-7368-4e48-8732-782df89e89dd-cert\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.365911 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-config-volume\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.365911 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365736 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-service-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.365911 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365765 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzzb\" (UniqueName: \"kubernetes.io/projected/e9cad0b0-2b17-48f6-9c14-12a419791b1f-kube-api-access-bdzzb\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.365911 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365783 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-config\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.365911 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.365803 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.865782701 +0000 UTC m=+32.556246274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:22.366168 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365945 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.366168 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.365947 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.366168 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.366168 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-stats-auth\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsd9\" (UniqueName: \"kubernetes.io/projected/44e3677f-537d-4cd9-8b72-3486e64246b6-kube-api-access-4wsd9\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5g9\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366278 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366306 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-tmp\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366318 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db4c92c3-fe37-4fbf-be54-a0196309c59c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:22.366373 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366359 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468a8f95-953e-403c-a2d7-931d530da8ee-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhk2\" (UniqueName: \"kubernetes.io/projected/fb4e8bad-6ee9-4270-a117-70210189abe1-kube-api-access-xhhk2\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e3677f-537d-4cd9-8b72-3486e64246b6-serving-cert\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-snapshots\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.366741 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.367079 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.367079 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vm2\" (UniqueName: \"kubernetes.io/projected/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-kube-api-access-29vm2\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.367079 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.367079 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb4e8bad-6ee9-4270-a117-70210189abe1-serving-cert\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.367079 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.366967 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.367079 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb926\" (UniqueName: \"kubernetes.io/projected/be3d02e8-cb60-4762-b79f-e29afc5b53ac-kube-api-access-qb926\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.367344 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.367344 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.367344 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367241 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-snapshots\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.367344 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-default-certificate\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.367344 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96z7w\" (UniqueName: \"kubernetes.io/projected/efec2af7-bad8-43d3-8fd8-45071136bdcf-kube-api-access-96z7w\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.367528 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.367528 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9cad0b0-2b17-48f6-9c14-12a419791b1f-serving-cert\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.367528 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.367528 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e3677f-537d-4cd9-8b72-3486e64246b6-config\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.367676 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn85\" (UniqueName: \"kubernetes.io/projected/ed820d3a-f16c-4bf0-a30e-85ad2732ab58-kube-api-access-rgn85\") pod \"kas-connection-checker-685895c7c9-9cdsm\" (UID: \"ed820d3a-f16c-4bf0-a30e-85ad2732ab58\") " pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" Oct 31 16:02:22.367676 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.367571 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:22.367676 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.367623 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.867607659 +0000 UTC m=+32.558071210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:22.367791 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.367791 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367712 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.367871 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.367791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.368047 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.368029 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e3677f-537d-4cd9-8b72-3486e64246b6-config\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.370420 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.370400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-stats-auth\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.370509 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.370462 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-default-certificate\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.391639 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.391602 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 16:02:22.391639 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.391603 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 16:02:22.391800 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.391644 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 16:02:22.392576 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392481 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 16:02:22.392576 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392521 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 16:02:22.392576 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392528 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 16:02:22.392576 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392554 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 16:02:22.392793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392650 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 16:02:22.392793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392664 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 16:02:22.392793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392679 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jwpwk\"" Oct 31 16:02:22.392793 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392788 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lhk9l\"" Oct 31 16:02:22.392941 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392815 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-85rcs\"" Oct 31 16:02:22.392941 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392837 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zwdm2\"" Oct 31 16:02:22.392941 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392855 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 16:02:22.392941 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392855 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 16:02:22.393068 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392887 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tp4vw\"" Oct 31 16:02:22.393068 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392888 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 16:02:22.393068 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.392908 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s8pkl\"" Oct 31 16:02:22.393445 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.393433 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m76mb"] Oct 31 16:02:22.394034 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.394023 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw"] Oct 31 16:02:22.394645 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.394631 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2gc9"] Oct 31 16:02:22.396082 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396068 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:22.396119 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396079 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:22.396119 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396095 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:22.396194 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396086 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:22.396194 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396181 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.896159808 +0000 UTC m=+32.586623368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:22.396268 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396200 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.896190615 +0000 UTC m=+32.586654178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:22.396792 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396767 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:22.396889 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396835 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.896819847 +0000 UTC m=+32.587283399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:22.396889 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396862 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:22.396889 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.396883 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.397062 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.396898 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.896889283 +0000 UTC m=+32.587352832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:22.397246 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.397224 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 16:02:22.397642 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.397478 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mwvsn\"" Oct 31 16:02:22.397755 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.397672 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 16:02:22.397818 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.397763 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 16:02:22.398604 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.398078 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dstd9\"" Oct 31 16:02:22.398604 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.398129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.398604 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.398244 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:22.398604 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.398315 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.898283993 +0000 UTC m=+32.588747558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:22.398604 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.398479 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sgkl6"] Oct 31 16:02:22.398604 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.398558 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.399061 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.399037 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e3677f-537d-4cd9-8b72-3486e64246b6-serving-cert\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.399229 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.399212 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.399326 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.399302 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.399551 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.399532 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb4e8bad-6ee9-4270-a117-70210189abe1-serving-cert\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.399599 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.399579 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.401254 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.401236 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.401254 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.401249 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9cad0b0-2b17-48f6-9c14-12a419791b1f-serving-cert\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.407977 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.407922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468a8f95-953e-403c-a2d7-931d530da8ee-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.408403 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.408387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.424517 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.424481 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.424668 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.424493 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6st7s\" (UniqueName: \"kubernetes.io/projected/b3d75aa1-14ee-4dd6-8216-440abcd707fa-kube-api-access-6st7s\") pod \"kas-connection-checker-685895c7c9-x72jc\" (UID: \"b3d75aa1-14ee-4dd6-8216-440abcd707fa\") " pod="kube-system/kas-connection-checker-685895c7c9-x72jc" Oct 31 16:02:22.424826 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.424806 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hm7f\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.424888 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.424819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzzb\" (UniqueName: \"kubernetes.io/projected/e9cad0b0-2b17-48f6-9c14-12a419791b1f-kube-api-access-bdzzb\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.425023 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.424994 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5p2\" (UniqueName: \"kubernetes.io/projected/468a8f95-953e-403c-a2d7-931d530da8ee-kube-api-access-2g5p2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.427219 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.427180 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgjb\" (UniqueName: \"kubernetes.io/projected/45d11ee0-d36c-4f4a-915c-fcedef344e68-kube-api-access-fpgjb\") pod \"kas-connection-checker-685895c7c9-69955\" (UID: \"45d11ee0-d36c-4f4a-915c-fcedef344e68\") " pod="kube-system/kas-connection-checker-685895c7c9-69955" Oct 31 16:02:22.427751 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.427725 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vm2\" (UniqueName: \"kubernetes.io/projected/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-kube-api-access-29vm2\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.427840 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.427787 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96z7w\" (UniqueName: \"kubernetes.io/projected/efec2af7-bad8-43d3-8fd8-45071136bdcf-kube-api-access-96z7w\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.428045 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.428022 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhk2\" (UniqueName: \"kubernetes.io/projected/fb4e8bad-6ee9-4270-a117-70210189abe1-kube-api-access-xhhk2\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.430694 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.428245 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.430694 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.428842 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb926\" (UniqueName: \"kubernetes.io/projected/be3d02e8-cb60-4762-b79f-e29afc5b53ac-kube-api-access-qb926\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.430694 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.429651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn85\" (UniqueName: \"kubernetes.io/projected/ed820d3a-f16c-4bf0-a30e-85ad2732ab58-kube-api-access-rgn85\") pod \"kas-connection-checker-685895c7c9-9cdsm\" (UID: \"ed820d3a-f16c-4bf0-a30e-85ad2732ab58\") " pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" Oct 31 16:02:22.430694 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.429781 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsd9\" (UniqueName: \"kubernetes.io/projected/44e3677f-537d-4cd9-8b72-3486e64246b6-kube-api-access-4wsd9\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.430694 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.430371 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5g9\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.431398 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.431381 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86bf6\" (UniqueName: \"kubernetes.io/projected/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222-kube-api-access-86bf6\") pod \"volume-data-source-validator-58c9f5f46d-xx5j8\" (UID: \"7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" Oct 31 16:02:22.468839 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.468802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e06bd6-7368-4e48-8732-782df89e89dd-cert\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.469010 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.468843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-config-volume\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.469010 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.468918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.469125 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.469011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-tmp-dir\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.469125 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.469040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqp5\" (UniqueName: \"kubernetes.io/projected/f5e06bd6-7368-4e48-8732-782df89e89dd-kube-api-access-5mqp5\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.469125 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.469069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sd5\" (UniqueName: \"kubernetes.io/projected/65899ffa-6ce0-4db1-8759-b17f99943f51-kube-api-access-j9sd5\") pod \"network-check-source-6f78d59f99-hlsdq\" (UID: \"65899ffa-6ce0-4db1-8759-b17f99943f51\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" Oct 31 16:02:22.469125 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.469098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8vd\" (UniqueName: \"kubernetes.io/projected/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-kube-api-access-mk8vd\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.469329 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.469097 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:22.469329 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.469256 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:22.969235865 +0000 UTC m=+32.659699427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:22.469411 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.469372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-tmp-dir\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.469480 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.469462 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-config-volume\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.470418 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.470402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" Oct 31 16:02:22.471118 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.471100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e06bd6-7368-4e48-8732-782df89e89dd-cert\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.475734 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.475713 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-x72jc" Oct 31 16:02:22.483194 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.481597 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-69955" Oct 31 16:02:22.510760 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.510728 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:22.526464 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.526437 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sd5\" (UniqueName: \"kubernetes.io/projected/65899ffa-6ce0-4db1-8759-b17f99943f51-kube-api-access-j9sd5\") pod \"network-check-source-6f78d59f99-hlsdq\" (UID: \"65899ffa-6ce0-4db1-8759-b17f99943f51\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" Oct 31 16:02:22.528338 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.528308 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8vd\" (UniqueName: \"kubernetes.io/projected/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-kube-api-access-mk8vd\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.529854 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.529825 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqp5\" (UniqueName: \"kubernetes.io/projected/f5e06bd6-7368-4e48-8732-782df89e89dd-kube-api-access-5mqp5\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.530819 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.530789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:22.537651 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.537623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" Oct 31 16:02:22.545858 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.545834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:22.553317 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.552979 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:22.583738 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.583497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" Oct 31 16:02:22.593907 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.593701 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:22.854183 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:22.854109 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d75aa1_14ee_4dd6_8216_440abcd707fa.slice/crio-36caba845c626ccbb2f80d79abaec9af9d9cf428788e4c08099903f83c512222 WatchSource:0}: Error finding container 36caba845c626ccbb2f80d79abaec9af9d9cf428788e4c08099903f83c512222: Status 404 returned error can't find the container with id 36caba845c626ccbb2f80d79abaec9af9d9cf428788e4c08099903f83c512222 Oct 31 16:02:22.854734 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.854714 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-x72jc"] Oct 31 16:02:22.862842 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.862821 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-69955"] Oct 31 16:02:22.863007 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:22.862987 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded820d3a_f16c_4bf0_a30e_85ad2732ab58.slice/crio-65c89198b5770281008484fc19e49c03a10127ae5577ab1ab5dc79d1dee38e65 WatchSource:0}: Error finding container 65c89198b5770281008484fc19e49c03a10127ae5577ab1ab5dc79d1dee38e65: Status 404 returned error can't find the container with id 65c89198b5770281008484fc19e49c03a10127ae5577ab1ab5dc79d1dee38e65 Oct 31 16:02:22.863371 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.863354 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9cdsm"] Oct 31 16:02:22.865027 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:22.865000 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d11ee0_d36c_4f4a_915c_fcedef344e68.slice/crio-b432a6dca8276e43d84ff3de0dc49bfc24d0d12c9d6aba0655a4bc8fe55013ad WatchSource:0}: Error finding container b432a6dca8276e43d84ff3de0dc49bfc24d0d12c9d6aba0655a4bc8fe55013ad: Status 404 returned error can't find the container with id b432a6dca8276e43d84ff3de0dc49bfc24d0d12c9d6aba0655a4bc8fe55013ad Oct 31 16:02:22.866464 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.866442 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:22.866567 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.866428 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:22.875957 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.875926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.876052 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.876037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:22.876105 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.876076 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:22.876188 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.876174 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.87615413 +0000 UTC m=+33.566617694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:22.876264 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.876249 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.876232724 +0000 UTC m=+33.566696288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:22.902544 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.902404 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-bthm6"] Oct 31 16:02:22.914231 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:22.914185 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cad0b0_2b17_48f6_9c14_12a419791b1f.slice/crio-bc33acc075b42c1d78d15bfc8148a800bf9954bc289a04724fff23c5c32226bd WatchSource:0}: Error finding container bc33acc075b42c1d78d15bfc8148a800bf9954bc289a04724fff23c5c32226bd: Status 404 returned error can't find the container with id bc33acc075b42c1d78d15bfc8148a800bf9954bc289a04724fff23c5c32226bd Oct 31 16:02:22.919475 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.919100 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 16:02:22.919475 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.919335 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 16:02:22.919917 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.919581 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4fqcg\"" Oct 31 16:02:22.919917 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.919771 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fd4\"" Oct 31 16:02:22.943704 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.943675 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9zgx8"] Oct 31 16:02:22.948823 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:22.948788 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4e8bad_6ee9_4270_a117_70210189abe1.slice/crio-071061a1ff0cc6fdf07ddf12b5bf11dc6638f687c87874e5cd9eb2ec9ebe7764 WatchSource:0}: Error finding container 071061a1ff0cc6fdf07ddf12b5bf11dc6638f687c87874e5cd9eb2ec9ebe7764: Status 404 returned error can't find the container with id 071061a1ff0cc6fdf07ddf12b5bf11dc6638f687c87874e5cd9eb2ec9ebe7764 Oct 31 16:02:22.972683 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.972662 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8"] Oct 31 16:02:22.977098 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.977075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:22.977241 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.977148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:22.977241 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.977203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:22.977241 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977229 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:22.977401 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977237 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:22.977401 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977296 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.977274483 +0000 UTC m=+33.667738049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:22.977401 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977317 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.977307231 +0000 UTC m=+33.667770785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:22.977401 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977327 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:22.977401 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977341 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:22.977401 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.977365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:22.977401 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977376 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.97736535 +0000 UTC m=+33.667828905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977424 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977435 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.977429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977462 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.977452967 +0000 UTC m=+33.667916533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.977493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977510 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977540 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.977531171 +0000 UTC m=+33.667994725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977597 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:22.977706 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:22.977630 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:23.977620297 +0000 UTC m=+33.668083868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:22.979314 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:22.979297 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d63f3bd_5f89_4a1d_9ee6_9e0bcb099222.slice/crio-9a27f6f98e8982df8f9d20c57b0d1093488e34434ba13c164f623f7399b7d53f WatchSource:0}: Error finding container 9a27f6f98e8982df8f9d20c57b0d1093488e34434ba13c164f623f7399b7d53f: Status 404 returned error can't find the container with id 9a27f6f98e8982df8f9d20c57b0d1093488e34434ba13c164f623f7399b7d53f Oct 31 16:02:22.982391 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:22.982367 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl"] Oct 31 16:02:23.038242 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:23.038211 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e06bd6_7368_4e48_8732_782df89e89dd.slice/crio-0943763db09fc98441e4579f2ca6dca7b02ed5173dbd43f3c27f912a057be00c WatchSource:0}: Error finding container 0943763db09fc98441e4579f2ca6dca7b02ed5173dbd43f3c27f912a057be00c: Status 404 returned error can't find the container with id 0943763db09fc98441e4579f2ca6dca7b02ed5173dbd43f3c27f912a057be00c Oct 31 16:02:23.040766 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:23.040731 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65899ffa_6ce0_4db1_8759_b17f99943f51.slice/crio-2bd9779ea517e974992905ce2efec7410a9b9f41987bff1e1f05da60e59c55fe WatchSource:0}: Error finding container 2bd9779ea517e974992905ce2efec7410a9b9f41987bff1e1f05da60e59c55fe: Status 404 returned error can't find the container with id 2bd9779ea517e974992905ce2efec7410a9b9f41987bff1e1f05da60e59c55fe Oct 31 16:02:23.069475 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.069443 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs"] Oct 31 16:02:23.070145 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.070116 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sgkl6"] Oct 31 16:02:23.070883 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.070867 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq"] Oct 31 16:02:23.087202 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.087162 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" event={"ID":"468a8f95-953e-403c-a2d7-931d530da8ee","Type":"ContainerStarted","Data":"ce3319dcad67399b5e5fc45f4bf4e23c3b1f6e1c1d71277cd493ca7cf602f99f"} Oct 31 16:02:23.088618 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.088595 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" event={"ID":"e9cad0b0-2b17-48f6-9c14-12a419791b1f","Type":"ContainerStarted","Data":"bc33acc075b42c1d78d15bfc8148a800bf9954bc289a04724fff23c5c32226bd"} Oct 31 16:02:23.089820 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.089798 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-69955" event={"ID":"45d11ee0-d36c-4f4a-915c-fcedef344e68","Type":"ContainerStarted","Data":"02ef50cb622157c59a184dec6d651cdded401409dda28137da7d90bcd787022f"} Oct 31 16:02:23.089914 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.089839 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-69955" event={"ID":"45d11ee0-d36c-4f4a-915c-fcedef344e68","Type":"ContainerStarted","Data":"b432a6dca8276e43d84ff3de0dc49bfc24d0d12c9d6aba0655a4bc8fe55013ad"} Oct 31 16:02:23.091519 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.091500 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" event={"ID":"ed820d3a-f16c-4bf0-a30e-85ad2732ab58","Type":"ContainerStarted","Data":"64730958a3d65e38f1f7f787b4968da3bda121b5a237fcb3c1614071c59bf3fb"} Oct 31 16:02:23.091587 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.091531 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" event={"ID":"ed820d3a-f16c-4bf0-a30e-85ad2732ab58","Type":"ContainerStarted","Data":"65c89198b5770281008484fc19e49c03a10127ae5577ab1ab5dc79d1dee38e65"} Oct 31 16:02:23.092753 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.092733 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" event={"ID":"65899ffa-6ce0-4db1-8759-b17f99943f51","Type":"ContainerStarted","Data":"2bd9779ea517e974992905ce2efec7410a9b9f41987bff1e1f05da60e59c55fe"} Oct 31 16:02:23.093871 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.093853 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" event={"ID":"7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222","Type":"ContainerStarted","Data":"9a27f6f98e8982df8f9d20c57b0d1093488e34434ba13c164f623f7399b7d53f"} Oct 31 16:02:23.094877 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.094859 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9zgx8" event={"ID":"fb4e8bad-6ee9-4270-a117-70210189abe1","Type":"ContainerStarted","Data":"071061a1ff0cc6fdf07ddf12b5bf11dc6638f687c87874e5cd9eb2ec9ebe7764"} Oct 31 16:02:23.095876 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.095857 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sgkl6" event={"ID":"f5e06bd6-7368-4e48-8732-782df89e89dd","Type":"ContainerStarted","Data":"0943763db09fc98441e4579f2ca6dca7b02ed5173dbd43f3c27f912a057be00c"} Oct 31 16:02:23.096774 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.096756 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" event={"ID":"44e3677f-537d-4cd9-8b72-3486e64246b6","Type":"ContainerStarted","Data":"55a7ad30cfa4e7c61d6f330fb52fb14e1d39b0192b075efe1f064ac46d39fea4"} Oct 31 16:02:23.098028 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.098009 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x72jc" event={"ID":"b3d75aa1-14ee-4dd6-8216-440abcd707fa","Type":"ContainerStarted","Data":"406c841335015fef3e3efa763f056a693e299b94fe501e8a586f9274ac8b44c0"} Oct 31 16:02:23.098128 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.098044 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-x72jc" event={"ID":"b3d75aa1-14ee-4dd6-8216-440abcd707fa","Type":"ContainerStarted","Data":"36caba845c626ccbb2f80d79abaec9af9d9cf428788e4c08099903f83c512222"} Oct 31 16:02:23.785112 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.785072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:23.785305 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.785128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:23.785305 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.785222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:23.785411 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.785392 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 16:02:23.785469 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.785456 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:55.785438942 +0000 UTC m=+65.475902506 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : secret "metrics-daemon-secret" not found Oct 31 16:02:23.787391 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.787368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:23.787648 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.787632 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:23.789381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.789362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:23.866690 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.866660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:23.886478 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.886449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:23.886593 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.886558 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:23.886638 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.886602 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.886585113 +0000 UTC m=+35.577048662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:23.886681 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.886659 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:23.886709 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.886698 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.886687248 +0000 UTC m=+35.577150797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:23.919214 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.919185 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jqwtc\"" Oct 31 16:02:23.922293 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.922272 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:23.987388 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.987351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:23.987529 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.987424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:23.987529 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.987464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:23.987529 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.987494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987539 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987545 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987570 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987603 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.987584536 +0000 UTC m=+35.678048102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987624 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.987614031 +0000 UTC m=+35.678077583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987628 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.987553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987625 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987677 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.98766302 +0000 UTC m=+35.678126575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:23.987690 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987688 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:23.988044 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987723 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.987709909 +0000 UTC m=+35.678173463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:23.988044 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:23.987746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:23.988044 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987817 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.987805882 +0000 UTC m=+35.678269437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:23.988044 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987853 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:23.988044 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987865 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:23.988044 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:23.987903 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:25.987890196 +0000 UTC m=+35.678353757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:24.175307 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:24.175243 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a66684b_1ce2_4892_b466_d95589fcf351.slice/crio-9716e57acf04cd4f2bfd11a7872b385e843ebe0a934910a6835ad085d0d6b3f6 WatchSource:0}: Error finding container 9716e57acf04cd4f2bfd11a7872b385e843ebe0a934910a6835ad085d0d6b3f6: Status 404 returned error can't find the container with id 9716e57acf04cd4f2bfd11a7872b385e843ebe0a934910a6835ad085d0d6b3f6 Oct 31 16:02:24.177293 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:24.176832 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mvztk"] Oct 31 16:02:24.287678 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:24.287639 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kddfn"] Oct 31 16:02:24.291653 ip-10-0-142-127 kubenswrapper[2573]: W1031 16:02:24.291624 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9fc385_70b6_442b_b076_0102e41c5fbd.slice/crio-7259d3155a1abef76535bf0c7708e9b0e4cd33bfdfcb25f81a78823cfe3b7ac2 WatchSource:0}: Error finding container 7259d3155a1abef76535bf0c7708e9b0e4cd33bfdfcb25f81a78823cfe3b7ac2: Status 404 returned error can't find the container with id 7259d3155a1abef76535bf0c7708e9b0e4cd33bfdfcb25f81a78823cfe3b7ac2 Oct 31 16:02:24.502459 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:24.502358 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" podStartSLOduration=400.502336928 podStartE2EDuration="6m40.502336928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:24.389362052 +0000 UTC m=+34.079825624" watchObservedRunningTime="2026-10-31 16:02:24.502336928 +0000 UTC m=+34.192800503" Oct 31 16:02:24.617652 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:24.617578 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-69955" podStartSLOduration=400.617559511 podStartE2EDuration="6m40.617559511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:24.504079705 +0000 UTC m=+34.194543277" watchObservedRunningTime="2026-10-31 16:02:24.617559511 +0000 UTC m=+34.308023084" Oct 31 16:02:25.111635 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:25.111596 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kddfn" event={"ID":"ec9fc385-70b6-442b-b076-0102e41c5fbd","Type":"ContainerStarted","Data":"7259d3155a1abef76535bf0c7708e9b0e4cd33bfdfcb25f81a78823cfe3b7ac2"} Oct 31 16:02:25.115732 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:25.115638 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mvztk" event={"ID":"5a66684b-1ce2-4892-b466-d95589fcf351","Type":"ContainerStarted","Data":"9716e57acf04cd4f2bfd11a7872b385e843ebe0a934910a6835ad085d0d6b3f6"} Oct 31 16:02:25.409060 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:25.409019 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:25.603889 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:25.603847 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:25.909734 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:25.909119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:25.909734 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:25.909342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:25.909734 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:25.909567 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:25.909734 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:25.909631 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:29.909613841 +0000 UTC m=+39.600077396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:25.909734 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:25.909702 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:29.909691489 +0000 UTC m=+39.600155043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:26.010835 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.010749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:26.010835 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.010811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.010846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.010877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.010885 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.010905 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.010933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.010959 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:30.010941372 +0000 UTC m=+39.701404940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.011019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:26.011067 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011062 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011112 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:30.011096993 +0000 UTC m=+39.701560546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011162 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011175 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011181 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011211 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:30.011198424 +0000 UTC m=+39.701661991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011228 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:30.011218925 +0000 UTC m=+39.701682481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011261 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011284 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011292 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:30.011280174 +0000 UTC m=+39.701743728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:26.011573 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:26.011321 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:30.011311495 +0000 UTC m=+39.701775058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:26.129004 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.128226 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" event={"ID":"7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222","Type":"ContainerStarted","Data":"cfb1183c2dae227e0dd1d6ea67547e609f58fdefbed872dbd6668e8af96c4ed8"} Oct 31 16:02:26.244053 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.243987 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-x72jc" podStartSLOduration=402.243965072 podStartE2EDuration="6m42.243965072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:24.617914811 +0000 UTC m=+34.308378384" watchObservedRunningTime="2026-10-31 16:02:26.243965072 +0000 UTC m=+35.934428645" Oct 31 16:02:26.244662 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.244619 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" podStartSLOduration=364.536525716 podStartE2EDuration="6m7.244606356s" totalImagesPullingTime="2.70808064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:26.240854603 +0000 UTC m=+35.931318175" watchObservedRunningTime="2026-10-31 16:02:26.244606356 +0000 UTC m=+35.935069930" Oct 31 16:02:26.344603 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:26.344456 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:29.141275 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:29.141248 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/0.log" Oct 31 16:02:29.141691 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:29.141298 2573 generic.go:356] "Generic (PLEG): container finished" podID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerID="6591d142a69d503d5b9125e522190759cef24600d17cbc8176099aea6fa60320" exitCode=255 Oct 31 16:02:29.141691 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:29.141364 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" event={"ID":"e9cad0b0-2b17-48f6-9c14-12a419791b1f","Type":"ContainerDied","Data":"6591d142a69d503d5b9125e522190759cef24600d17cbc8176099aea6fa60320"} Oct 31 16:02:29.141803 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:29.141721 2573 scope.go:122] "RemoveContainer" containerID="6591d142a69d503d5b9125e522190759cef24600d17cbc8176099aea6fa60320" Oct 31 16:02:29.958874 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:29.958393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:29.958874 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:29.958469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:29.958874 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:29.958566 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:29.958874 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:29.958606 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.95858913 +0000 UTC m=+47.649052694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:29.958874 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:29.958622 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.95861611 +0000 UTC m=+47.649079659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.059588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.059669 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.059721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.059796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.059839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.059868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.059990 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060051 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.06003333 +0000 UTC m=+47.750496889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060114 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060172 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.060158364 +0000 UTC m=+47.750621916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060234 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060266 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.06025567 +0000 UTC m=+47.750719228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060330 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060340 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060371 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.060362512 +0000 UTC m=+47.750826067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060426 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:30.060589 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060435 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:30.061595 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060464 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.060453087 +0000 UTC m=+47.750916652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:30.061595 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060520 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:30.061595 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.060551 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:38.060541665 +0000 UTC m=+47.751005219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:30.150446 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.150413 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9zgx8" event={"ID":"fb4e8bad-6ee9-4270-a117-70210189abe1","Type":"ContainerStarted","Data":"5260060bac58f587fa611531df51b58e35a8996271a2ab2cbf7c8c4b4d5f5d88"} Oct 31 16:02:30.154167 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.153187 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sgkl6" event={"ID":"f5e06bd6-7368-4e48-8732-782df89e89dd","Type":"ContainerStarted","Data":"a54052165abc431b575dbb101570983b924ac516303f38231679904e3ab5aaef"} Oct 31 16:02:30.158214 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.158184 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" event={"ID":"44e3677f-537d-4cd9-8b72-3486e64246b6","Type":"ContainerStarted","Data":"d2b2b1099e9bf45b1faa743c1b64df77eac320008c776e8debca4a5a2ee6fb8a"} Oct 31 16:02:30.161803 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.161778 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" event={"ID":"468a8f95-953e-403c-a2d7-931d530da8ee","Type":"ContainerStarted","Data":"3151abc0569db732c20a7841e9bec5237e213d3910f245e58a2e077cbe9927fd"} Oct 31 16:02:30.164434 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.164419 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/1.log" Oct 31 16:02:30.164918 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.164900 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/0.log" Oct 31 16:02:30.164970 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.164935 2573 generic.go:356] "Generic (PLEG): container finished" podID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerID="ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732" exitCode=255 Oct 31 16:02:30.165005 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.164975 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" event={"ID":"e9cad0b0-2b17-48f6-9c14-12a419791b1f","Type":"ContainerDied","Data":"ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732"} Oct 31 16:02:30.165058 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.165016 2573 scope.go:122] "RemoveContainer" containerID="6591d142a69d503d5b9125e522190759cef24600d17cbc8176099aea6fa60320" Oct 31 16:02:30.165393 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.165379 2573 scope.go:122] "RemoveContainer" containerID="ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732" Oct 31 16:02:30.165575 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:30.165554 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:30.265847 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.265795 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-9zgx8" podStartSLOduration=365.158074745 podStartE2EDuration="6m11.265781164s" totalImagesPullingTime="6.107706419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:30.265130859 +0000 UTC m=+39.955594433" watchObservedRunningTime="2026-10-31 16:02:30.265781164 +0000 UTC m=+39.956244735" Oct 31 16:02:30.623277 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.623187 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sgkl6" podStartSLOduration=1.773570688 podStartE2EDuration="8.623170119s" totalImagesPullingTime="6.849599431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:30.49468522 +0000 UTC m=+40.185148792" watchObservedRunningTime="2026-10-31 16:02:30.623170119 +0000 UTC m=+40.313633690" Oct 31 16:02:30.668013 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.667980 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:30.745181 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.745107 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" podStartSLOduration=374.653955915 podStartE2EDuration="6m20.745087724s" totalImagesPullingTime="6.091131809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:30.629125323 +0000 UTC m=+40.319588895" watchObservedRunningTime="2026-10-31 16:02:30.745087724 +0000 UTC m=+40.435551297" Oct 31 16:02:30.745502 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.745471 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" podStartSLOduration=374.682505064 podStartE2EDuration="6m20.745462389s" totalImagesPullingTime="6.062957325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:30.744632862 +0000 UTC m=+40.435096433" watchObservedRunningTime="2026-10-31 16:02:30.745462389 +0000 UTC m=+40.435925962" Oct 31 16:02:30.775591 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.774356 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vs98d"] Oct 31 16:02:30.803327 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.803299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:30.815079 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.815014 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:30.831832 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.831806 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vs98d"] Oct 31 16:02:30.856019 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.855992 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 16:02:30.856019 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.855992 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 16:02:30.856279 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.856213 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7g99\"" Oct 31 16:02:30.968159 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.968104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jbs\" (UniqueName: \"kubernetes.io/projected/68b75f93-390b-4813-9264-a9b9709fc330-kube-api-access-q6jbs\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:30.968357 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.968330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b75f93-390b-4813-9264-a9b9709fc330-crio-socket\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:30.968492 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.968466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:30.968612 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.968531 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:30.968612 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.968564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b75f93-390b-4813-9264-a9b9709fc330-data-volume\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:30.980564 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:30.980530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:31.069720 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.069688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.069890 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.069745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.069890 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.069766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b75f93-390b-4813-9264-a9b9709fc330-data-volume\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.070014 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.069886 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 16:02:31.070014 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.069917 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 16:02:31.070014 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.069950 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:31.569933763 +0000 UTC m=+41.260397315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : configmap "kube-rbac-proxy" not found Oct 31 16:02:31.070014 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.069980 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:31.569962734 +0000 UTC m=+41.260426303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : secret "insights-runtime-extractor-tls" not found Oct 31 16:02:31.070014 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.069918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jbs\" (UniqueName: \"kubernetes.io/projected/68b75f93-390b-4813-9264-a9b9709fc330-kube-api-access-q6jbs\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.070257 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.070080 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b75f93-390b-4813-9264-a9b9709fc330-data-volume\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.070257 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.070099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b75f93-390b-4813-9264-a9b9709fc330-crio-socket\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.070257 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.070200 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b75f93-390b-4813-9264-a9b9709fc330-crio-socket\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.126553 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.126512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:31.128055 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.128026 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jbs\" (UniqueName: \"kubernetes.io/projected/68b75f93-390b-4813-9264-a9b9709fc330-kube-api-access-q6jbs\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.170567 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.170535 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/1.log" Oct 31 16:02:31.170998 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.170984 2573 scope.go:122] "RemoveContainer" containerID="ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732" Oct 31 16:02:31.171222 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.171192 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:31.172592 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.172559 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" event={"ID":"65899ffa-6ce0-4db1-8759-b17f99943f51","Type":"ContainerStarted","Data":"1c26dec042731f9ecbefa89799197fdee27f1353be1dde9e1c892a1aa50ade64"} Oct 31 16:02:31.265369 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.265286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:31.398045 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.397998 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" podStartSLOduration=363.688171296 podStartE2EDuration="6m11.397982832s" totalImagesPullingTime="7.709811536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:31.39757192 +0000 UTC m=+41.088035492" watchObservedRunningTime="2026-10-31 16:02:31.397982832 +0000 UTC m=+41.088446425" Oct 31 16:02:31.574760 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.574688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.574760 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:31.574748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:31.574954 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.574826 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 16:02:31.574954 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.574853 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 16:02:31.574954 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.574904 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:32.574884395 +0000 UTC m=+42.265347948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : configmap "kube-rbac-proxy" not found Oct 31 16:02:31.574954 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:31.574924 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:32.574914475 +0000 UTC m=+42.265378037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : secret "insights-runtime-extractor-tls" not found Oct 31 16:02:32.176551 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.176511 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kddfn" event={"ID":"ec9fc385-70b6-442b-b076-0102e41c5fbd","Type":"ContainerStarted","Data":"8d14b283bf599cceb73ed1486abb32e1cbce8ddc80bb5f1fd3fcfbe50fca8a5c"} Oct 31 16:02:32.177010 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.176670 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:32.290631 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.290582 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kddfn" podStartSLOduration=35.495000945 podStartE2EDuration="42.290566419s" totalImagesPullingTime="6.795565474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:32.289328224 +0000 UTC m=+41.979791796" watchObservedRunningTime="2026-10-31 16:02:32.290566419 +0000 UTC m=+41.981029991" Oct 31 16:02:32.324587 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.324554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:32.479606 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.479535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:32.511972 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.511946 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:32.512123 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.512002 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:32.512381 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.512365 2573 scope.go:122] "RemoveContainer" containerID="ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732" Oct 31 16:02:32.512577 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:32.512555 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:32.583618 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.583590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:32.583725 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.583636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:32.583766 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:32.583744 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 16:02:32.583766 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:32.583758 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 16:02:32.583847 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:32.583813 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.58379249 +0000 UTC m=+44.274256042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : secret "insights-runtime-extractor-tls" not found Oct 31 16:02:32.583847 ip-10-0-142-127 kubenswrapper[2573]: E1031 16:02:32.583832 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.583823484 +0000 UTC m=+44.274287047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : configmap "kube-rbac-proxy" not found Oct 31 16:02:32.855211 ip-10-0-142-127 kubenswrapper[2573]: I1031 16:02:32.855110 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:32.864993 ip-10-0-142-127 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 16:02:32.879607 ip-10-0-142-127 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 16:02:32.879800 ip-10-0-142-127 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 16:02:32.879851 ip-10-0-142-127 systemd[1]: kubelet.service: Consumed 3.064s CPU time. Oct 31 16:02:32.894777 ip-10-0-142-127 systemd[1]: Starting Kubernetes Kubelet... Oct 31 16:02:32.995921 ip-10-0-142-127 kubenswrapper[4880]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 16:02:32.995921 ip-10-0-142-127 kubenswrapper[4880]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 16:02:32.995921 ip-10-0-142-127 kubenswrapper[4880]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 16:02:32.995921 ip-10-0-142-127 kubenswrapper[4880]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 16:02:32.998073 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.997976 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:32.998073 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998072 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:32.998073 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998077 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998081 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998085 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998087 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998091 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998094 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998096 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998099 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998102 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998105 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998108 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998111 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998113 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998116 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998118 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998121 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998125 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998128 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998130 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:32.998190 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998148 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998153 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998157 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998160 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998163 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998166 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998168 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998177 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998180 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998183 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998185 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998188 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998191 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998193 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998196 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998199 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998202 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998205 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998207 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:32.998909 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998210 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998212 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998215 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998218 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998220 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998223 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998226 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998228 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998230 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998233 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998235 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998238 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998241 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998243 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998245 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998249 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998252 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998255 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998258 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998260 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:32.999398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998263 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998266 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998269 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998271 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998274 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998276 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998279 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998281 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998284 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998286 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998289 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998291 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998294 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998296 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998300 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998302 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998305 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998308 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998310 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998314 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:32.999892 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998318 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998321 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998324 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998327 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998329 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998332 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998335 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998338 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998341 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998343 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998346 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998348 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998351 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998353 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998356 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998360 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998364 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998366 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998369 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998372 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.000402 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998374 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998377 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998379 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998382 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998384 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.998387 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999236 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999247 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999251 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999254 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999257 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999260 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999262 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999265 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999268 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999270 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999273 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999277 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999281 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:33.000896 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999285 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999288 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999291 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999294 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999296 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999299 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999302 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999305 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999308 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999312 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999314 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999317 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999320 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999322 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999325 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999328 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999330 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999333 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999337 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:33.001395 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999340 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999345 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999349 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999352 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999356 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999359 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999362 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999366 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999369 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999371 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999374 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999377 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999380 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999383 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999386 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999388 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999391 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999394 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999396 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999399 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:33.001865 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999402 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999405 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999409 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999412 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999414 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999417 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999419 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999422 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999426 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999428 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999431 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999433 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999436 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999440 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999442 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999445 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999447 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999450 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999453 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999455 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:33.002366 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999458 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999461 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999463 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999466 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999469 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999473 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999475 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999478 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999480 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999483 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999485 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999488 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999490 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999493 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999496 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999499 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999502 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999504 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999506 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:33.002863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999509 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999512 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999514 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999517 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999519 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999522 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999526 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999528 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999531 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999533 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999536 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999538 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999541 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999543 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:32.999546 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000065 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000071 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000074 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000077 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000080 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:33.003398 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000083 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000086 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000089 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000092 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000094 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000097 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000100 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000103 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000106 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000108 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000111 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000114 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000116 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000119 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000121 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000124 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000126 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000129 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000148 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000151 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:33.003886 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000153 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000156 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000160 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000163 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000165 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000168 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000171 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000173 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000176 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000178 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000181 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000184 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000188 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000192 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000195 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000198 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000200 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000203 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000206 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:33.004568 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000208 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000211 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000214 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000217 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000220 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000222 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000225 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000228 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000231 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000234 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000237 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000241 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000244 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000247 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000249 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000252 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000255 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000257 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000260 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000263 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:33.005345 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000265 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000268 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000270 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000273 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000276 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000280 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000283 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000285 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000288 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000290 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000293 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000295 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000298 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000301 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000303 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000307 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000309 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000312 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000314 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:33.005857 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000317 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000319 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000322 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000324 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000329 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000332 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000335 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000338 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000340 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000344 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000346 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000349 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000352 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000354 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000357 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000360 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000362 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000365 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000367 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:33.006355 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000370 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000372 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000375 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.000377 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000478 4880 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000488 4880 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000497 4880 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000504 4880 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000510 4880 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000515 4880 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000524 4880 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000528 4880 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000531 4880 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000535 4880 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000538 4880 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000542 4880 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000545 4880 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000550 4880 flags.go:64] FLAG: --cgroup-root="" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000553 4880 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000556 4880 flags.go:64] FLAG: --client-ca-file="" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000559 4880 flags.go:64] FLAG: --cloud-provider="external" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000562 4880 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000567 4880 flags.go:64] FLAG: --cluster-domain="" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000570 4880 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 16:02:33.006841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000573 4880 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000576 4880 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000580 4880 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000585 4880 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000588 4880 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000592 4880 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000595 4880 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000598 4880 flags.go:64] FLAG: --contention-profiling="false" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000601 4880 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000604 4880 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000608 4880 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000611 4880 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000616 4880 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000619 4880 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000622 4880 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000625 4880 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000628 4880 flags.go:64] FLAG: --enable-server="true" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000632 4880 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000637 4880 flags.go:64] FLAG: --event-burst="100" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000641 4880 flags.go:64] FLAG: --event-qps="50" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000644 4880 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000647 4880 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000650 4880 flags.go:64] FLAG: --eviction-hard="" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000655 4880 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 16:02:33.007446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000658 4880 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000663 4880 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000666 4880 flags.go:64] FLAG: --eviction-soft="" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000670 4880 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000673 4880 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000676 4880 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000679 4880 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000682 4880 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000685 4880 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000688 4880 flags.go:64] FLAG: --feature-gates="" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000692 4880 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000695 4880 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000699 4880 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000703 4880 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000706 4880 flags.go:64] FLAG: --healthz-port="10248" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000709 4880 flags.go:64] FLAG: --help="false" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000712 4880 flags.go:64] FLAG: --hostname-override="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000716 4880 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000719 4880 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000722 4880 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000725 4880 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000729 4880 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000732 4880 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 16:02:33.008060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000735 4880 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000738 4880 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000741 4880 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000744 4880 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000747 4880 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000750 4880 flags.go:64] FLAG: --kube-reserved="" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000754 4880 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000757 4880 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000760 4880 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000763 4880 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000769 4880 flags.go:64] FLAG: --lock-file="" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000772 4880 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000776 4880 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000779 4880 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000784 4880 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000787 4880 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000791 4880 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000794 4880 flags.go:64] FLAG: --logging-format="text" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000797 4880 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000800 4880 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000803 4880 flags.go:64] FLAG: --manifest-url="" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000806 4880 flags.go:64] FLAG: --manifest-url-header="" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000811 4880 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000815 4880 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000819 4880 flags.go:64] FLAG: --max-pods="110" Oct 31 16:02:33.008660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000822 4880 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000825 4880 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000828 4880 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000831 4880 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000834 4880 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000837 4880 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000841 4880 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000848 4880 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000851 4880 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000854 4880 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000857 4880 flags.go:64] FLAG: --pod-cidr="" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000860 4880 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000864 4880 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000867 4880 flags.go:64] FLAG: --pods-per-core="0" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000871 4880 flags.go:64] FLAG: --port="10250" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000874 4880 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000877 4880 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a53437074682e059" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000882 4880 flags.go:64] FLAG: --qos-reserved="" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000885 4880 flags.go:64] FLAG: --read-only-port="10255" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000888 4880 flags.go:64] FLAG: --register-node="true" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000891 4880 flags.go:64] FLAG: --register-with-taints="" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000895 4880 flags.go:64] FLAG: --registry-burst="10" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000898 4880 flags.go:64] FLAG: --registry-qps="5" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000902 4880 flags.go:64] FLAG: --reserved-cpus="" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000905 4880 flags.go:64] FLAG: --reserved-memory="" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000909 4880 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 16:02:33.009327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000912 4880 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000915 4880 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000919 4880 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000922 4880 flags.go:64] FLAG: --runonce="false" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000925 4880 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000929 4880 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000932 4880 flags.go:64] FLAG: --seccomp-default="false" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000935 4880 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000938 4880 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000941 4880 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000944 4880 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000948 4880 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000959 4880 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000963 4880 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000966 4880 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000969 4880 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000973 4880 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000976 4880 flags.go:64] FLAG: --system-cgroups="" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000979 4880 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000985 4880 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000988 4880 flags.go:64] FLAG: --tls-cert-file="" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000992 4880 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.000996 4880 flags.go:64] FLAG: --tls-min-version="" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001000 4880 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001003 4880 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 16:02:33.010060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001008 4880 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001011 4880 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001014 4880 flags.go:64] FLAG: --v="2" Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001019 4880 flags.go:64] FLAG: --version="false" Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001023 4880 flags.go:64] FLAG: --vmodule="" Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001027 4880 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001031 4880 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001180 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001184 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001187 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001190 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001192 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001195 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001197 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001200 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001202 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001205 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001209 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001213 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001216 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001219 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:33.010717 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001222 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001224 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001227 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001230 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001232 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001236 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001238 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001245 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001248 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001251 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001254 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001257 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001259 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001262 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001264 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001267 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001269 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001272 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001275 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001277 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.011284 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001280 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001283 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001286 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001290 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001293 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001297 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001300 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001302 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001305 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001308 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001310 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001313 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001316 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001319 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001321 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001324 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001326 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001329 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001332 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.011858 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001335 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001338 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001341 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001343 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001346 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001349 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001351 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001354 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001357 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001360 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001362 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001365 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001367 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001370 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001373 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001375 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001378 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001380 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001383 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001385 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:33.012621 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001388 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001391 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001393 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001396 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001399 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001402 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001405 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001407 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001410 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001413 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001416 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001418 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001421 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001423 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001426 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001429 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001433 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001436 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001438 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:33.013281 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001441 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001444 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001447 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001450 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001452 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001455 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001457 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001460 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001463 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001466 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001468 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001471 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001473 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001476 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.001504 4880 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001540 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:33.013863 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001544 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001548 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001550 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001553 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001555 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001558 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001560 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001563 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001566 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001569 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001571 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001574 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001577 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001580 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001584 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001587 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001589 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001592 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001595 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001597 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:33.014438 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001600 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001602 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001605 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001608 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001610 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001613 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001615 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001618 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001621 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001623 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001626 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001629 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001631 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001634 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001636 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001639 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001642 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001644 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001647 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:33.014928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001651 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001654 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001656 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001659 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001661 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001664 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001667 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001671 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001675 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001679 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001683 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001686 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001689 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001692 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001695 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001698 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001701 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001704 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001707 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001709 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:33.015585 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001713 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001716 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001719 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001721 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001724 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001727 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001729 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001732 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001735 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001737 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001740 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001742 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001747 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001750 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001754 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001757 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001760 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001762 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001765 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.016126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001770 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001773 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001775 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001778 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001781 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001784 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001786 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001789 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001791 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001794 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001796 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001799 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001801 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001804 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001806 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001809 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001811 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001814 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001817 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:33.016993 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001819 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001822 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001825 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001828 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001831 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001833 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001837 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.001840 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.006949 4880 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.006969 4880 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.006996 4880 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.007001 4880 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007065 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007070 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007073 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007077 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007079 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007083 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007086 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:33.017795 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007088 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007091 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007094 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007097 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007100 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007102 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007105 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007109 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007112 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007115 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007118 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007120 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007123 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007126 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007128 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007131 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007148 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007153 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007157 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:33.018657 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007161 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007166 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007169 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007172 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007175 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007177 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007180 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007183 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007186 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007191 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007195 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007198 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007201 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007204 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007207 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007210 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007213 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007216 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007218 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:33.019448 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007221 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007224 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007227 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007230 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007232 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007235 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007237 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007240 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007243 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007245 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007248 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007251 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007254 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007258 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007262 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007265 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007267 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007270 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007273 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007276 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:33.020298 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007278 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007281 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007283 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007286 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007289 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007292 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007294 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007297 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007300 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007303 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007305 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007309 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007311 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007314 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007317 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007320 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007323 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007325 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007328 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:33.021027 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007331 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007333 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007336 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007338 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007342 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007345 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007347 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007350 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007352 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007355 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007357 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007360 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007363 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007366 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007368 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007371 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007374 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007376 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007379 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007381 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:33.021651 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007384 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007387 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.007417 4880 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007588 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007593 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007596 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007599 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007602 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007605 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007608 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007610 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007613 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007616 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007619 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007623 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007627 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:33.022164 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007630 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007633 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007635 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007638 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007641 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007643 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007646 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007648 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007651 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007654 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007656 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007659 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007661 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007664 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007666 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007669 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007671 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007674 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007677 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:33.022556 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007680 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007682 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007685 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007687 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007690 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007693 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007696 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007699 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007701 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007704 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007707 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007709 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007711 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007715 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007717 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007720 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007723 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007725 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007728 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007730 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:33.023042 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007733 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007736 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007738 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007741 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007743 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007746 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007749 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007751 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007754 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007756 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007759 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007762 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007765 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007768 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007770 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007774 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007778 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007781 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007784 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:33.023572 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007792 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007795 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007797 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007800 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007803 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007806 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007808 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007811 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007813 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007816 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007819 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007821 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007824 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007826 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007829 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007831 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007834 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007836 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007839 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007841 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:33.024026 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007844 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007846 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007849 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007851 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007854 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007857 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007859 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007862 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007865 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007867 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007870 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007873 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007875 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007883 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007886 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.007909 4880 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 16:02:33.024529 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007942 4880 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007945 4880 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007948 4880 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007951 4880 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007954 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007957 4880 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007959 4880 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007962 4880 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007965 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007968 4880 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007971 4880 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007973 4880 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007976 4880 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007978 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007981 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007984 4880 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007986 4880 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007989 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007992 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 16:02:33.025012 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007996 4880 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.007999 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008003 4880 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008006 4880 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008009 4880 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008012 4880 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008015 4880 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008018 4880 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008020 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008023 4880 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008026 4880 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008029 4880 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008032 4880 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008034 4880 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008037 4880 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008039 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008042 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008045 4880 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008047 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008050 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 16:02:33.025514 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008052 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008055 4880 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008057 4880 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008060 4880 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008062 4880 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008065 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008068 4880 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008070 4880 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008073 4880 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008077 4880 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008079 4880 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008082 4880 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008085 4880 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008088 4880 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008090 4880 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008093 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008096 4880 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008098 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008102 4880 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008104 4880 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 16:02:33.026003 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008107 4880 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008109 4880 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008112 4880 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008115 4880 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008118 4880 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008121 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008123 4880 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008126 4880 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008129 4880 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008131 4880 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008159 4880 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008163 4880 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008166 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008169 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008172 4880 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008174 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008177 4880 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008179 4880 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008182 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 16:02:33.026510 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008185 4880 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008187 4880 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008190 4880 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008192 4880 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008195 4880 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008198 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008200 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008206 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008208 4880 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008211 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008214 4880 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008217 4880 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008219 4880 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008222 4880 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008225 4880 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008227 4880 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008231 4880 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008234 4880 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008237 4880 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 16:02:33.027006 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008240 4880 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008242 4880 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008245 4880 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008248 4880 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008250 4880 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008253 4880 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008256 4880 feature_gate.go:386] unrecognized feature gate: Example Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008258 4880 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:33.008261 4880 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.008327 4880 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.009422 4880 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.009497 4880 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.009698 4880 server.go:1014] "Starting client certificate rotation" Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.009859 4880 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.009966 4880 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 15:56:50 +0000 UTC" deadline="2028-04-07 02:55:24.022082038 +0000 UTC" Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.010008 4880 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12562h52m51.01207705s" Oct 31 16:02:33.027486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.010233 4880 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 16:02:33.027887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.011404 4880 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 16:02:33.027887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.013246 4880 log.go:25] "Validated CRI v1 runtime API" Oct 31 16:02:33.027887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.015087 4880 log.go:25] "Validated CRI v1 image API" Oct 31 16:02:33.027887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.015827 4880 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 16:02:33.027887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.018760 4880 fs.go:135] Filesystem UUIDs: map[20011175-f2a4-4c46-9bd1-9b4d239776e9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e3fc3432-3a15-4341-a7c4-9ddea9cbe2cf:/dev/nvme0n1p3] Oct 31 16:02:33.028035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.018790 4880 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/071061a1ff0cc6fdf07ddf12b5bf11dc6638f687c87874e5cd9eb2ec9ebe7764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/071061a1ff0cc6fdf07ddf12b5bf11dc6638f687c87874e5cd9eb2ec9ebe7764/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0943763db09fc98441e4579f2ca6dca7b02ed5173dbd43f3c27f912a057be00c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0943763db09fc98441e4579f2ca6dca7b02ed5173dbd43f3c27f912a057be00c/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c398d0815dd3229bb73cf811951de1b35f48e67093b9f0b33f46cf305d20313/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c398d0815dd3229bb73cf811951de1b35f48e67093b9f0b33f46cf305d20313/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bd9779ea517e974992905ce2efec7410a9b9f41987bff1e1f05da60e59c55fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bd9779ea517e974992905ce2efec7410a9b9f41987bff1e1f05da60e59c55fe/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36caba845c626ccbb2f80d79abaec9af9d9cf428788e4c08099903f83c512222/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36caba845c626ccbb2f80d79abaec9af9d9cf428788e4c08099903f83c512222/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/538c4450eb30426950288b75c8b5a200cbf34315f2068917609e0fa204cb08ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/538c4450eb30426950288b75c8b5a200cbf34315f2068917609e0fa204cb08ff/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55a7ad30cfa4e7c61d6f330fb52fb14e1d39b0192b075efe1f064ac46d39fea4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55a7ad30cfa4e7c61d6f330fb52fb14e1d39b0192b075efe1f064ac46d39fea4/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65c89198b5770281008484fc19e49c03a10127ae5577ab1ab5dc79d1dee38e65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65c89198b5770281008484fc19e49c03a10127ae5577ab1ab5dc79d1dee38e65/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c92a3bb8e43c0f1642447fdd8761494e77e3363a51003629f23d30add3a8722/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c92a3bb8e43c0f1642447fdd8761494e77e3363a51003629f23d30add3a8722/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7259d3155a1abef76535bf0c7708e9b0e4cd33bfdfcb25f81a78823cfe3b7ac2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7259d3155a1abef76535bf0c7708e9b0e4cd33bfdfcb25f81a78823cfe3b7ac2/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72d608b8c51f0f4b7739bcbbfb265a18ff4e44729982f6dd8d718e81f3941986/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72d608b8c51f0f4b7739bcbbfb265a18ff4e44729982f6dd8d718e81f3941986/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79eeb9f1f1d45b72bce1e68044be92c18d92f91de780725c1058e95269a70379/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79eeb9f1f1d45b72bce1e68044be92c18d92f91de780725c1058e95269a70379/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9716e57acf04cd4f2bfd11a7872b385e843ebe0a934910a6835ad085d0d6b3f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9716e57acf04cd4f2bfd11a7872b385e843ebe0a934910a6835ad085d0d6b3f6/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a27f6f98e8982df8f9d20c57b0d1093488e34434ba13c164f623f7399b7d53f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a27f6f98e8982df8f9d20c57b0d1093488e34434ba13c164f623f7399b7d53f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b432a6dca8276e43d84ff3de0dc49bfc24d0d12c9d6aba0655a4bc8fe55013ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b432a6dca8276e43d84ff3de0dc49bfc24d0d12c9d6aba0655a4bc8fe55013ad/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba3ec2f618b10d953c518daac66851775f06f8a32bf325dd2b61dda04195ac2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba3ec2f618b10d953c518daac66851775f06f8a32bf325dd2b61dda04195ac2c/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc33acc075b42c1d78d15bfc8148a800bf9954bc289a04724fff23c5c32226bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc33acc075b42c1d78d15bfc8148a800bf9954bc289a04724fff23c5c32226bd/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7fb3ca298b02979ab13cf08f753b95ba974fcc728ed61d74042de4de444e1b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7fb3ca298b02979ab13cf08f753b95ba974fcc728ed61d74042de4de444e1b0/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce3319dcad67399b5e5fc45f4bf4e23c3b1f6e1c1d71277cd493ca7cf602f99f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce3319dcad67399b5e5fc45f4bf4e23c3b1f6e1c1d71277cd493ca7cf602f99f/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea6406e6f83cc72bc1721251e525217ab0c964d4145bc634e18f96dce94e5cac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea6406e6f83cc72bc1721251e525217ab0c964d4145bc634e18f96dce94e5cac/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d948a90-6dab-4c2e-b2a3-6815efc5b06e/volumes/kubernetes.io~projected/kube-api-access-mk8vd:{mountpoint:/var/lib/kubelet/pods/0d948a90-6dab-4c2e-b2a3-6815efc5b06e/volumes/kubernetes.io~projected/kube-api-access-mk8vd major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37a8427f-f59a-4c8e-8c1a-67791b68f4a5/volumes/kubernetes.io~projected/kube-api-access-cqs6c:{mountpoint:/var/lib/kubelet/pods/37a8427f-f59a-4c8e-8c1a-67791b68f4a5/volumes/kubernetes.io~projected/kube-api-access-cqs6c major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~projected/kube-api-access-6hm7f:{mountpoint:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~projected/kube-api-access-6hm7f major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43a99cec-f100-473d-b831-f8052ae72bf0/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/43a99cec-f100-473d-b831-f8052ae72bf0/volumes/kubernetes.io~secret/agent-certs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44c3511b-f377-4032-a196-2490d71549ae/volumes/kubernetes.io~projected/kube-api-access-plhvw:{mountpoint:/var/lib/kubelet/pods/44c3511b-f377-4032-a196-2490d71549ae/volumes/kubernetes.io~projected/kube-api-access-plhvw major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e3677f-537d-4cd9-8b72-3486e64246b6/volumes/kubernetes.io~projected/kube-api-access-4wsd9:{mountpoint:/var/lib/kubelet/pods/44e3677f-537d-4cd9-8b72-3486e64246b6/volumes/kubernetes.io~projected/kube-api-access-4wsd9 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44e3677f-537d-4cd9-8b72-3486e64246b6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/44e3677f-537d-4cd9-8b72-3486e64246b6/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45d11ee0-d36c-4f4a-915c-fcedef344e68/volumes/kubernetes.io~projected/kube-api-access-fpgjb:{mountpoint:/var/lib/kubelet/pods/45d11ee0-d36c-4f4a-915c-fcedef344e68/volumes/kubernetes.io~projected/kube-api-access-fpgjb major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/468a8f95-953e-403c-a2d7-931d530da8ee/volumes/kubernetes.io~projected/kube-api-access-2g5p2:{mountpoint:/var/lib/kubelet/pods/468a8f95-953e-403c-a2d7-931d530da8ee/volumes/kubernetes.io~projected/kube-api-access-2g5p2 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/468a8f95-953e-403c-a2d7-931d530da8ee/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/468a8f95-953e-403c-a2d7-931d530da8ee/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122/volumes/kubernetes.io~projected/kube-api-access-29vm2:{mountpoint:/var/lib/kubelet/pods/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122/volumes/kubernetes.io~projected/kube-api-access-29vm2 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55a6396f-e3a0-42ec-88fa-c4d23384e807/volumes/kubernetes.io~projected/kube-api-access-wrm9r:{mountpoint:/var/lib/kubelet/pods/55a6396f-e3a0-42ec-88fa-c4d23384e807/volumes/kubernetes.io~projected/kube-api-access-wrm9r major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a66684b-1ce2-4892-b466-d95589fcf351/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5a66684b-1ce2-4892-b466-d95589fcf351/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~projected/kube-api-access-vb5g9:{mountpoint:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~projected/kube-api-access-vb5g9 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad/volumes/kubernetes.io~projected/kube-api-access-78wzd:{mountpoint:/var/lib/kubelet/pods/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad/volumes/kubernetes.io~projected/kube-api-access-78wzd major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65899ffa-6ce0-4db1-8759-b17f99943f51/volumes/kubernetes.io~projected/kube-api-access-j9sd5:{mountpoint:/var/lib/kubelet/pods/65899ffa-6ce0-4db1-8759-b17f99943f51/volumes/kubernetes.io~projected/kube-api-access-j9sd5 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68b75f93-390b-4813-9264-a9b9709fc330/volumes/kubernetes.io~projected/kube-api-access-q6jbs:{mountpoint:/var/lib/kubelet/pods/68b75f93-390b-4813-9264-a9b9709fc330/volumes/kubernetes.io~projected/kube-api-access-q6jbs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222/volumes/kubernetes.io~projected/kube-api-access-86bf6:{mountpoint:/var/lib/kubelet/pods/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222/volumes/kubernetes.io~projected/kube-api-access-86bf6 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~projected/kube-api-access-8tpvx:{mountpoint:/var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~projected/kube-api-access-8tpvx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95a4cede-89fb-421e-be64-69f3ea494179/volumes/kubernetes.io~projected/kube-api-access-5k9ll:{mountpoint:/var/lib/kubelet/pods/95a4cede-89fb-421e-be64-69f3ea494179/volumes/kubernetes.io~projected/kube-api-access-5k9ll major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb6cc68-a2af-438f-83be-9f1e9afc57b6/volumes/kubernetes.io~projected/kube-api-access-6pgj6:{mountpoint:/var/lib/kubelet/pods/9cb6cc68-a2af-438f-83be-9f1e9afc57b6/volumes/kubernetes.io~projected/kube-api-access-6pgj6 major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d75aa1-14ee-4dd6-8216-440abcd707fa/volumes/kubernetes.io~projected/kube-api-access-6st7s:{mountpoint:/var/lib/kubelet/pods/b3d75aa1-14ee-4dd6-8216-440abcd707fa/volumes/kubernetes.io~projected/kube-api-access-6st7s major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be3d02e8-cb60-4762-b79f-e29afc5b53ac/volumes/kubernetes.io~projected/kube-api-access-qb926:{mountpoint:/var/lib/kubelet/pods/be3d02e8-cb60-4762-b79f-e29afc5b53ac/volumes/kubernetes.io~projected/kube-api-access-qb926 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volumes/kubernetes.io~projected/kube-api-access-45vjp:{mountpoint:/var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volumes/kubernetes.io~projected/kube-api-access-45vjp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9cad0b0-2b17-48f6-9c14-12a419791b1f/volumes/kubernetes.io~projected/kube-api-access-bdzzb:{mountpoint:/var/lib/kubelet/pods/e9cad0b0-2b17-48f6-9c14-12a419791b1f/volumes/kubernetes.io~projected/kube-api-access-bdzzb major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9cad0b0-2b17-48f6-9c14-12a419791b1f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e9cad0b0-2b17-48f6-9c14-12a419791b1f/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb9982b4-6251-4984-8493-e884e03f13ae/volumes/kubernetes.io~projected/kube-api-access-9nbvr:{mountpoint:/var/lib/kubelet/pods/eb9982b4-6251-4984-8493-e884e03f13ae/volumes/kubernetes.io~projected/kube-api-access-9nbvr major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec9fc385-70b6-442b-b076-0102e41c5fbd/volumes/kubernetes.io~projected/kube-api-access-qbg9n:{mountpoint:/var/lib/kubelet/pods/ec9fc385-70b6-442b-b076-0102e41c5fbd/volumes/kubernetes.io~projected/kube-api-access-qbg9n major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed820d3a-f16c-4bf0-a30e-85ad2732ab58/volumes/kubernetes.io~projected/kube-api-access-rgn85:{mountpoint:/var/lib/kubelet/pods/ed820d3a-f16c-4bf0-a30e-85ad2732ab58/volumes/kubernetes.io~projected/kube-api-access-rgn85 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~projected/kube-api-access-96z7w:{mountpoint:/var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~projected/kube-api-access-96z7w major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5e06bd6-7368-4e48-8732-782df89e89dd/volumes/kubernetes.io~projected/kube-api-access-5mqp5:{mountpoint:/var/lib/kubelet/pods/f5e06bd6-7368-4e48-8732-782df89e89dd/volumes/kubernetes.io~projected/kube-api-access-5mqp5 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5e06bd6-7368-4e48-8732-782df89e89dd/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f5e06bd6-7368-4e48-8732-782df89e89dd/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb4e8bad-6ee9-4270-a117-70210189abe1/volumes/kubernetes.io~projected/kube-api-access-xhhk2:{mountpoint:/var/lib/kubelet/pods/fb4e8bad-6ee9-4270-a117-70210189abe1/volumes/kubernetes.io~projected/kube-api-access-xhhk2 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb4e8bad-6ee9-4270-a117-70210189abe1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fb4e8bad-6ee9-4270-a117-70210189abe1/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/094294917e535d1a262c1531cd5d258022bbbad03b789d557f5180a1a2b2b5d8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/833c743a2395298e29e952840d5f4818f7195bb7a6068b0950cc0ad7a4506d40/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/554796d4a7d428ace0c0c72616c1a3e41fc06f3105aad3d4d0a7e23ed59ea4b5/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/48dec7796e6f4658b9cedf11e56c94307227cd26d86babac2485d82c6b06e0f2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/a48d77bd53a81eefcfa2c074bef90174a1c6ac77fa535c91f219a8ae3c01fae5/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c16bdc1f3397d3be9d92f0fc17c6830dca2f4d34367c995ecc230cf8cbb49f3a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f016bb1543c8a9f8821a3f3e96d19cfd164e260135f30f2f22eed1cb8b176c98/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e5f9dcb4216015c103c1ca2e8dba5ab79f4b9e92d79a43fb2bc46055fcea0a35/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1f2c24d0d1f4564cf3c46d101816496df98c338798559d080f78a823841721a7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/3615b2a27a8f2130a3cd5c7269d835b22bbd5f48b5525d8e1a65230417e0c5b2/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/a7885b3c3dd975a1dc04c4fd185d00d9b8170be3547c0550d1acbe6dd7b1244b/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/efdd556a07a2eb1de86f6dbf63353b8fb8f46efe25cca39bc7aacc009c85a5b9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d6cc1d953d53d7c89fe7ae98d59fd5c9f844b4db2f212b632a51347249f1b5f0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b190d4807e592d00e48c605a901111bc03ffe80d500646e84e66805be8d24c18/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a544b5b6bed1d5aa94b2084d18d08ce9b83a0947b64025b26be87b246677088d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/98c60ebbed51bac798a6ee22b9921f5b9571132920d3121be4bb865f8a0c2114/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/81da9d3879430ecf4adb3efbd7a4a2fb8495ac5300eecda65817bf6b17765d2d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/54c2a57b60064d167c65fde4cd776792c3ba537948309a54f9152a256f623b34/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/fa63de1adc9d73c91e55703dd35afadf92f8ce5a96f7e4bdc1b21a83130ead31/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f1b49742049a704d0870cd14b725f4d7e8f91b2962fc1b3c68c83712b4f9ab3a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7cd7411c23d6dfae498c1f8d6fdec300c6d65fc834d157d4f0db327dc72b2b6a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/64b7e1546f87f7e661a93a4391b1fe779cb5751a4a333e699d4b7e75d0704f6f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f72cd3221844a278d7c62a8489855b67b17183fd13b0a932a98f22a332864fbe/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/fde17e0cf986c9fd9392a79acb3a7e5d835cac838606f7f5d8cdec191f8456b9/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/718edf921102b59c53b7a39be1930b8d625f559b86108bb7000f655fe96e4d25/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/92ee057bf998819ee51f87c682a040e0859a79ea3663a30c3759188973e1d467/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/89ff384cad5565a5c6ab9737b4bddba98791f6ad021d5fb772e6f2277529da1c/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/68978c96e2fa6bd3a1cabd98dfea36b53a8bd8fab6be17f3c87c7b5965c2ee74/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/28f4489edb3ba178cc47e6523957aea4176281be27562415be038875f2e23779/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/525be706d46fe82a5f93577c064c5c1e606aadf592ead5ca00cf12fe2f5c1ce4/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/59aabf33ec3f5054a6b47c7fd07f9f485de5f1e677d20374e357bf68395d68c9/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/2262fa50e32c805119ef70f8f81f1c7271df6efd745212fe8029aa7ac36bf0c5/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/e5751ecc705b41d27227d1f4e49c20d3a12629e9daab3bedfe25613ee06f3740/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/750cf92796e4a622064fd4aa3d83e39e152486f6817f0f0de0738751683cf905/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/9c39835178338a0188183444b206ee3eb70ff21fb163b2f82f12642c7ccb5223/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/f70d3496960a8d14dd5058a1251edd3fdf89104cd0338c12a5aa940161d68e13/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/e23fdd6570ee75b3ad28932aac730d5d09b66c308983e745f95b39acad367723/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/6bda57339e976382cd3821c3ebe7e12c100c5d302fe73f8c6ba6766319f0d91e/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/abfba3ce21c8395d7eb447ebbd1c96f3976cc1376be588a91741cfbc47b897ff/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/de0b84f2b1d3e8a4c329124fb388a8f3ede818ff1e27674700bed9ed5376618e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/d427777fdef5a0299466b3b2e188d10fdbb84c8cf8a1e072fadd14d788f66246/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/8cddd590ecd77fc8ef13ceab8dfc20cd02f89bc317c5516b6a0bbb9679162e81/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/094eee5471ddc4f2200efce1f00e95737c486c0f94b46d668547bc9bcc11b857/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/c9858f220b9bb14e8d71e97a0181472ecec6c90f6aaded53e6a68c38aaab2427/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/14bcc38debddd19e7d2fc9f6fb38a252a15fbbd9d4eed8fd22fe1c02ba48af00/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/6d660fe732e5bec482ba513323ecdb24ffc7faef97cab67c7a710d7f0b98d953/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d344c9b2b63dad238dcbc678b5162fd90e1d3c9f103b08c7b03f6b8083cfc43f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/d048fbfca6c5faebd0e133711594229483e58f85f882ca70e00589c8f7c2ae1f/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/29bfbdd51f5c06f1b7210ffe499cf11b58e57de1d2a958abc84a9c9f7508d9eb/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f3e9194a5046ff784a9bf9ed6b04c6bb0b889e89f8bf14d702caeb371c66cf68/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a610d3ffeed2b3a25e01b58a4e9ecc67d450507616c125687b5b993e7933dcc1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/5120304116d9a20dbf6b376931439a89fee616f90b73d890a82c410bd6660591/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/67f24c1ed06b96bd20b821d64c5311a73662e31833c2f0fa8a4b065224ff9cbf/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1c4f4520333a308162f606acd1b189d2a5e74e9b11f161752a8ba603b81ee162/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 16:02:33.029947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.029523 4880 manager.go:217] Machine: {Timestamp:2026-10-31 16:02:33.028997439 +0000 UTC m=+0.064292295 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099400 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26b5f2cef135b6e421564abb14eda7 SystemUUID:ec26b5f2-cef1-35b6-e421-564abb14eda7 BootID:34639023-6b40-4003-a846-8c34e47aaaca Filesystems:[{Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc33acc075b42c1d78d15bfc8148a800bf9954bc289a04724fff23c5c32226bd/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~projected/kube-api-access-8tpvx DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79eeb9f1f1d45b72bce1e68044be92c18d92f91de780725c1058e95269a70379/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea6406e6f83cc72bc1721251e525217ab0c964d4145bc634e18f96dce94e5cac/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba3ec2f618b10d953c518daac66851775f06f8a32bf325dd2b61dda04195ac2c/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a27f6f98e8982df8f9d20c57b0d1093488e34434ba13c164f623f7399b7d53f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44e3677f-537d-4cd9-8b72-3486e64246b6/volumes/kubernetes.io~projected/kube-api-access-4wsd9 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bd9779ea517e974992905ce2efec7410a9b9f41987bff1e1f05da60e59c55fe/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95a4cede-89fb-421e-be64-69f3ea494179/volumes/kubernetes.io~projected/kube-api-access-5k9ll DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/be3d02e8-cb60-4762-b79f-e29afc5b53ac/volumes/kubernetes.io~projected/kube-api-access-qb926 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volumes/kubernetes.io~projected/kube-api-access-45vjp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~projected/kube-api-access-vb5g9 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9716e57acf04cd4f2bfd11a7872b385e843ebe0a934910a6835ad085d0d6b3f6/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb9982b4-6251-4984-8493-e884e03f13ae/volumes/kubernetes.io~projected/kube-api-access-9nbvr DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9cad0b0-2b17-48f6-9c14-12a419791b1f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44c3511b-f377-4032-a196-2490d71549ae/volumes/kubernetes.io~projected/kube-api-access-plhvw DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a66684b-1ce2-4892-b466-d95589fcf351/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68b75f93-390b-4813-9264-a9b9709fc330/volumes/kubernetes.io~projected/kube-api-access-q6jbs DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36caba845c626ccbb2f80d79abaec9af9d9cf428788e4c08099903f83c512222/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad/volumes/kubernetes.io~projected/kube-api-access-78wzd DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37a8427f-f59a-4c8e-8c1a-67791b68f4a5/volumes/kubernetes.io~projected/kube-api-access-cqs6c DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72d608b8c51f0f4b7739bcbbfb265a18ff4e44729982f6dd8d718e81f3941986/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~projected/kube-api-access-6hm7f DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/55a6396f-e3a0-42ec-88fa-c4d23384e807/volumes/kubernetes.io~projected/kube-api-access-wrm9r DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222/volumes/kubernetes.io~projected/kube-api-access-86bf6 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b432a6dca8276e43d84ff3de0dc49bfc24d0d12c9d6aba0655a4bc8fe55013ad/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9cad0b0-2b17-48f6-9c14-12a419791b1f/volumes/kubernetes.io~projected/kube-api-access-bdzzb DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f5e06bd6-7368-4e48-8732-782df89e89dd/volumes/kubernetes.io~projected/kube-api-access-5mqp5 DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65c89198b5770281008484fc19e49c03a10127ae5577ab1ab5dc79d1dee38e65/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/468a8f95-953e-403c-a2d7-931d530da8ee/volumes/kubernetes.io~projected/kube-api-access-2g5p2 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/45d11ee0-d36c-4f4a-915c-fcedef344e68/volumes/kubernetes.io~projected/kube-api-access-fpgjb DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/071061a1ff0cc6fdf07ddf12b5bf11dc6638f687c87874e5cd9eb2ec9ebe7764/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55a7ad30cfa4e7c61d6f330fb52fb14e1d39b0192b075efe1f064ac46d39fea4/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/538c4450eb30426950288b75c8b5a200cbf34315f2068917609e0fa204cb08ff/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d75aa1-14ee-4dd6-8216-440abcd707fa/volumes/kubernetes.io~projected/kube-api-access-6st7s DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/910df55b-a9db-4aa6-acc1-3721c868beb8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7fb3ca298b02979ab13cf08f753b95ba974fcc728ed61d74042de4de444e1b0/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed820d3a-f16c-4bf0-a30e-85ad2732ab58/volumes/kubernetes.io~projected/kube-api-access-rgn85 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb4e8bad-6ee9-4270-a117-70210189abe1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/468a8f95-953e-403c-a2d7-931d530da8ee/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65899ffa-6ce0-4db1-8759-b17f99943f51/volumes/kubernetes.io~projected/kube-api-access-j9sd5 DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5e06bd6-7368-4e48-8732-782df89e89dd/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7259d3155a1abef76535bf0c7708e9b0e4cd33bfdfcb25f81a78823cfe3b7ac2/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c398d0815dd3229bb73cf811951de1b35f48e67093b9f0b33f46cf305d20313/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce3319dcad67399b5e5fc45f4bf4e23c3b1f6e1c1d71277cd493ca7cf602f99f/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c92a3bb8e43c0f1642447fdd8761494e77e3363a51003629f23d30add3a8722/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb4e8bad-6ee9-4270-a117-70210189abe1/volumes/kubernetes.io~projected/kube-api-access-xhhk2 DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0943763db09fc98441e4579f2ca6dca7b02ed5173dbd43f3c27f912a057be00c/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec9fc385-70b6-442b-b076-0102e41c5fbd/volumes/kubernetes.io~projected/kube-api-access-qbg9n DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb6cc68-a2af-438f-83be-9f1e9afc57b6/volumes/kubernetes.io~projected/kube-api-access-6pgj6 DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43a99cec-f100-473d-b831-f8052ae72bf0/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d948a90-6dab-4c2e-b2a3-6815efc5b06e/volumes/kubernetes.io~projected/kube-api-access-mk8vd DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce582737-4c20-4d14-9199-ef952d229b5a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44e3677f-537d-4cd9-8b72-3486e64246b6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122/volumes/kubernetes.io~projected/kube-api-access-29vm2 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/efec2af7-bad8-43d3-8fd8-45071136bdcf/volumes/kubernetes.io~projected/kube-api-access-96z7w DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:071061a1ff0cc6f MacAddress:f6:56:53:96:6f:cc Speed:10000 Mtu:8901} {Name:0943763db09fc98 MacAddress:36:d3:f8:87:05:d9 Speed:10000 Mtu:8901} {Name:2bd9779ea517e97 MacAddress:fa:69:22:4b:c7:d4 Speed:10000 Mtu:8901} {Name:36caba845c626cc MacAddress:f2:04:31:aa:a2:9f Speed:10000 Mtu:8901} {Name:55a7ad30cfa4e7c MacAddress:aa:e2:69:f0:d9:2e Speed:10000 Mtu:8901} {Name:65c89198b577028 MacAddress:42:8a:1c:c7:c7:9f Speed:10000 Mtu:8901} {Name:7259d3155a1abef MacAddress:f6:26:05:2f:24:20 Speed:10000 Mtu:8901} {Name:9716e57acf04cd4 MacAddress:7e:ec:b1:0e:40:26 Speed:10000 Mtu:8901} {Name:9a27f6f98e8982d MacAddress:02:48:41:9c:9a:ca Speed:10000 Mtu:8901} {Name:b432a6dca8276e4 MacAddress:ca:4d:12:a9:ba:bd Speed:10000 Mtu:8901} {Name:bc33acc075b42c1 MacAddress:86:9b:8a:89:8b:ae Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cc:28:1e:25 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:1b:a8:34:e6:c4 Speed:0 Mtu:8901} {Name:ce3319dcad67399 MacAddress:8a:49:39:72:78:19 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:cc:28:1e:25 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:26:eb:1d:ed:01:46 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:dc:13:2e:2b:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 16:02:33.029947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.029949 4880 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 16:02:33.030207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030009 4880 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 16:02:33.030424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030395 4880 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 16:02:33.030559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030424 4880 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-127.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 16:02:33.030603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030561 4880 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 16:02:33.030603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030571 4880 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 16:02:33.030603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030582 4880 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 16:02:33.030686 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030603 4880 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 16:02:33.030686 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030623 4880 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 16:02:33.030837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030827 4880 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 16:02:33.030963 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030954 4880 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 16:02:33.031002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.030994 4880 kubelet.go:493] "Attempting to sync node with API server" Oct 31 16:02:33.031035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.031008 4880 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 16:02:33.031035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.031018 4880 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 16:02:33.031035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.031028 4880 kubelet.go:405] "Adding apiserver pod source" Oct 31 16:02:33.031152 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.031035 4880 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 16:02:33.032057 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.032037 4880 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 16:02:33.032057 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.032054 4880 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 16:02:33.032794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.032778 4880 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 16:02:33.035551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.035537 4880 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 16:02:33.035551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.035551 4880 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 16:02:33.035676 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.035666 4880 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 16:02:33.035879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.035867 4880 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 16:02:33.035909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.035898 4880 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 16:02:33.036023 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036013 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 16:02:33.036055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036029 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 16:02:33.036055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036036 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 16:02:33.036055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036042 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 16:02:33.036055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036051 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036061 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036075 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036082 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036088 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036094 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036107 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036117 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036158 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 16:02:33.036172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.036167 4880 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 16:02:33.039901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.039883 4880 server.go:1263] "Started kubelet" Oct 31 16:02:33.040015 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.039938 4880 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 16:02:33.041162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.040566 4880 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 16:02:33.041162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.040668 4880 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 16:02:33.041162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.041025 4880 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 16:02:33.041520 ip-10-0-142-127 systemd[1]: Started Kubernetes Kubelet. Oct 31 16:02:33.043427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.042565 4880 server.go:317] "Adding debug handlers to kubelet server" Oct 31 16:02:33.044822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.044530 4880 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 16:02:33.044822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.044526 4880 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 16:02:33.045735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.045665 4880 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 15:56:51 +0000 UTC" deadline="2028-06-23 01:48:04.005631966 +0000 UTC" Oct 31 16:02:33.045735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.045729 4880 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14409h45m30.959906644s" Oct 31 16:02:33.047715 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.045897 4880 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" Oct 31 16:02:33.047715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.046043 4880 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 16:02:33.047715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.046060 4880 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 16:02:33.047715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.046231 4880 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 16:02:33.047715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.047127 4880 factory.go:55] Registering systemd factory Oct 31 16:02:33.047715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.047165 4880 factory.go:223] Registration of the systemd container factory successfully Oct 31 16:02:33.048050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.048013 4880 factory.go:153] Registering CRI-O factory Oct 31 16:02:33.048050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.048036 4880 factory.go:223] Registration of the crio container factory successfully Oct 31 16:02:33.048168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.048125 4880 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 16:02:33.048223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.048173 4880 factory.go:103] Registering Raw factory Oct 31 16:02:33.048223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.048194 4880 manager.go:1199] Started watching for new ooms in manager Oct 31 16:02:33.048595 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.048579 4880 manager.go:319] Starting recovery of all containers Oct 31 16:02:33.049060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.048998 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40737053-5dda-4724-be16-bcdeca25cfec" volumeName="kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration" seLinuxMountContext="" Oct 31 16:02:33.049165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049066 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a99cec-f100-473d-b831-f8052ae72bf0" volumeName="kubernetes.io/secret/43a99cec-f100-473d-b831-f8052ae72bf0-agent-certs" seLinuxMountContext="" Oct 31 16:02:33.049165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049083 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6483e8c0-5b7a-46c6-b270-1f741dd0e8ad" volumeName="kubernetes.io/projected/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-kube-api-access-78wzd" seLinuxMountContext="" Oct 31 16:02:33.049165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049099 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="468a8f95-953e-403c-a2d7-931d530da8ee" volumeName="kubernetes.io/projected/468a8f95-953e-403c-a2d7-931d530da8ee-kube-api-access-2g5p2" seLinuxMountContext="" Oct 31 16:02:33.049165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049116 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222" volumeName="kubernetes.io/projected/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222-kube-api-access-86bf6" seLinuxMountContext="" Oct 31 16:02:33.049165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049130 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb9982b4-6251-4984-8493-e884e03f13ae" volumeName="kubernetes.io/projected/eb9982b4-6251-4984-8493-e884e03f13ae-kube-api-access-9nbvr" seLinuxMountContext="" Oct 31 16:02:33.049165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049158 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" volumeName="kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049170 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" volumeName="kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049185 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b75f93-390b-4813-9264-a9b9709fc330" volumeName="kubernetes.io/empty-dir/68b75f93-390b-4813-9264-a9b9709fc330-data-volume" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049199 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68b75f93-390b-4813-9264-a9b9709fc330" volumeName="kubernetes.io/projected/68b75f93-390b-4813-9264-a9b9709fc330-kube-api-access-q6jbs" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049211 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be3d02e8-cb60-4762-b79f-e29afc5b53ac" volumeName="kubernetes.io/configmap/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-config" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049226 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d948a90-6dab-4c2e-b2a3-6815efc5b06e" volumeName="kubernetes.io/configmap/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-config-volume" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049238 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43a99cec-f100-473d-b831-f8052ae72bf0" volumeName="kubernetes.io/configmap/43a99cec-f100-473d-b831-f8052ae72bf0-konnectivity-ca" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049252 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910df55b-a9db-4aa6-acc1-3721c868beb8" volumeName="kubernetes.io/projected/910df55b-a9db-4aa6-acc1-3721c868beb8-kube-api-access-8tpvx" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049304 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb6cc68-a2af-438f-83be-9f1e9afc57b6" volumeName="kubernetes.io/configmap/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-serviceca" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049328 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db4c92c3-fe37-4fbf-be54-a0196309c59c" volumeName="kubernetes.io/configmap/db4c92c3-fe37-4fbf-be54-a0196309c59c-nginx-conf" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049343 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec9fc385-70b6-442b-b076-0102e41c5fbd" volumeName="kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049359 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d948a90-6dab-4c2e-b2a3-6815efc5b06e" volumeName="kubernetes.io/projected/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-kube-api-access-mk8vd" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049373 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40737053-5dda-4724-be16-bcdeca25cfec" volumeName="kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049386 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95a4cede-89fb-421e-be64-69f3ea494179" volumeName="kubernetes.io/empty-dir/95a4cede-89fb-421e-be64-69f3ea494179-tmp-dir" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049399 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce582737-4c20-4d14-9199-ef952d229b5a" volumeName="kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-config" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049412 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb4e8bad-6ee9-4270-a117-70210189abe1" volumeName="kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-service-ca-bundle" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049426 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb4e8bad-6ee9-4270-a117-70210189abe1" volumeName="kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-snapshots" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049442 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40737053-5dda-4724-be16-bcdeca25cfec" volumeName="kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049455 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3677f-537d-4cd9-8b72-3486e64246b6" volumeName="kubernetes.io/projected/44e3677f-537d-4cd9-8b72-3486e64246b6-kube-api-access-4wsd9" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049469 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45d11ee0-d36c-4f4a-915c-fcedef344e68" volumeName="kubernetes.io/projected/45d11ee0-d36c-4f4a-915c-fcedef344e68-kube-api-access-fpgjb" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049499 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" volumeName="kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted" seLinuxMountContext="" Oct 31 16:02:33.049507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049513 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6483e8c0-5b7a-46c6-b270-1f741dd0e8ad" volumeName="kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-daemon-config" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049542 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed820d3a-f16c-4bf0-a30e-85ad2732ab58" volumeName="kubernetes.io/projected/ed820d3a-f16c-4bf0-a30e-85ad2732ab58-kube-api-access-rgn85" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049559 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="468a8f95-953e-403c-a2d7-931d530da8ee" volumeName="kubernetes.io/secret/468a8f95-953e-403c-a2d7-931d530da8ee-serving-cert" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049572 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" volumeName="kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049601 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95a4cede-89fb-421e-be64-69f3ea494179" volumeName="kubernetes.io/projected/95a4cede-89fb-421e-be64-69f3ea494179-kube-api-access-5k9ll" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049617 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce582737-4c20-4d14-9199-ef952d229b5a" volumeName="kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-env-overrides" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049629 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb4e8bad-6ee9-4270-a117-70210189abe1" volumeName="kubernetes.io/projected/fb4e8bad-6ee9-4270-a117-70210189abe1-kube-api-access-xhhk2" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049641 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" volumeName="kubernetes.io/projected/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-kube-api-access-cqs6c" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049657 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55a6396f-e3a0-42ec-88fa-c4d23384e807" volumeName="kubernetes.io/projected/55a6396f-e3a0-42ec-88fa-c4d23384e807-kube-api-access-wrm9r" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049670 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce582737-4c20-4d14-9199-ef952d229b5a" volumeName="kubernetes.io/secret/ce582737-4c20-4d14-9199-ef952d229b5a-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049683 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" volumeName="kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-binary-copy" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049696 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44c3511b-f377-4032-a196-2490d71549ae" volumeName="kubernetes.io/projected/44c3511b-f377-4032-a196-2490d71549ae-kube-api-access-plhvw" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049718 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" volumeName="kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049730 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" volumeName="kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049742 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40737053-5dda-4724-be16-bcdeca25cfec" volumeName="kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049757 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40737053-5dda-4724-be16-bcdeca25cfec" volumeName="kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049769 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa7d16a-1eba-4ab3-88f6-d6ae9c638122" volumeName="kubernetes.io/projected/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-kube-api-access-29vm2" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049782 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" volumeName="kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049799 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910df55b-a9db-4aa6-acc1-3721c868beb8" volumeName="kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-tmp" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049811 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9cad0b0-2b17-48f6-9c14-12a419791b1f" volumeName="kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-trusted-ca" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049823 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d948a90-6dab-4c2e-b2a3-6815efc5b06e" volumeName="kubernetes.io/empty-dir/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-tmp-dir" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049835 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40737053-5dda-4724-be16-bcdeca25cfec" volumeName="kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049848 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efec2af7-bad8-43d3-8fd8-45071136bdcf" volumeName="kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-stats-auth" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049859 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb4e8bad-6ee9-4270-a117-70210189abe1" volumeName="kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-trusted-ca-bundle" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049871 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40737053-5dda-4724-be16-bcdeca25cfec" volumeName="kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049883 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3677f-537d-4cd9-8b72-3486e64246b6" volumeName="kubernetes.io/configmap/44e3677f-537d-4cd9-8b72-3486e64246b6-config" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049897 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6483e8c0-5b7a-46c6-b270-1f741dd0e8ad" volumeName="kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cni-binary-copy" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049911 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65899ffa-6ce0-4db1-8759-b17f99943f51" volumeName="kubernetes.io/projected/65899ffa-6ce0-4db1-8759-b17f99943f51-kube-api-access-j9sd5" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049924 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be3d02e8-cb60-4762-b79f-e29afc5b53ac" volumeName="kubernetes.io/projected/be3d02e8-cb60-4762-b79f-e29afc5b53ac-kube-api-access-qb926" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049947 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9cad0b0-2b17-48f6-9c14-12a419791b1f" volumeName="kubernetes.io/projected/e9cad0b0-2b17-48f6-9c14-12a419791b1f-kube-api-access-bdzzb" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049960 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb9982b4-6251-4984-8493-e884e03f13ae" volumeName="kubernetes.io/configmap/eb9982b4-6251-4984-8493-e884e03f13ae-iptables-alerter-script" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049973 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb4e8bad-6ee9-4270-a117-70210189abe1" volumeName="kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-tmp" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049987 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" volumeName="kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.049999 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" volumeName="kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050013 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb6cc68-a2af-438f-83be-9f1e9afc57b6" volumeName="kubernetes.io/projected/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-kube-api-access-6pgj6" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050025 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce582737-4c20-4d14-9199-ef952d229b5a" volumeName="kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-script-lib" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050038 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9cad0b0-2b17-48f6-9c14-12a419791b1f" volumeName="kubernetes.io/secret/e9cad0b0-2b17-48f6-9c14-12a419791b1f-serving-cert" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050050 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5e06bd6-7368-4e48-8732-782df89e89dd" volumeName="kubernetes.io/projected/f5e06bd6-7368-4e48-8732-782df89e89dd-kube-api-access-5mqp5" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050064 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa7d16a-1eba-4ab3-88f6-d6ae9c638122" volumeName="kubernetes.io/configmap/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-telemetry-config" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050076 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d75aa1-14ee-4dd6-8216-440abcd707fa" volumeName="kubernetes.io/projected/b3d75aa1-14ee-4dd6-8216-440abcd707fa-kube-api-access-6st7s" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050088 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5e06bd6-7368-4e48-8732-782df89e89dd" volumeName="kubernetes.io/secret/f5e06bd6-7368-4e48-8732-782df89e89dd-cert" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050100 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb4e8bad-6ee9-4270-a117-70210189abe1" volumeName="kubernetes.io/secret/fb4e8bad-6ee9-4270-a117-70210189abe1-serving-cert" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050114 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="910df55b-a9db-4aa6-acc1-3721c868beb8" volumeName="kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-tuned" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050125 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9cad0b0-2b17-48f6-9c14-12a419791b1f" volumeName="kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-config" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050155 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efec2af7-bad8-43d3-8fd8-45071136bdcf" volumeName="kubernetes.io/projected/efec2af7-bad8-43d3-8fd8-45071136bdcf-kube-api-access-96z7w" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050168 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a66684b-1ce2-4892-b466-d95589fcf351" volumeName="kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050179 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44e3677f-537d-4cd9-8b72-3486e64246b6" volumeName="kubernetes.io/secret/44e3677f-537d-4cd9-8b72-3486e64246b6-serving-cert" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050192 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="468a8f95-953e-403c-a2d7-931d530da8ee" volumeName="kubernetes.io/configmap/468a8f95-953e-403c-a2d7-931d530da8ee-config" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050205 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce582737-4c20-4d14-9199-ef952d229b5a" volumeName="kubernetes.io/projected/ce582737-4c20-4d14-9199-ef952d229b5a-kube-api-access-45vjp" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050217 4880 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efec2af7-bad8-43d3-8fd8-45071136bdcf" volumeName="kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-default-certificate" seLinuxMountContext="" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050229 4880 reconstruct.go:97] "Volume reconstruction finished" Oct 31 16:02:33.051271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.050237 4880 reconciler.go:29] "Reconciler: start to sync state" Oct 31 16:02:33.067551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.067516 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/1.log" Oct 31 16:02:33.071265 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.071236 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/1.log" Oct 31 16:02:33.081009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.080965 4880 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 16:02:33.082314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.082292 4880 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 16:02:33.082314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.082318 4880 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 16:02:33.082513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.082354 4880 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 16:02:33.082513 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.082397 4880 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 16:02:33.107294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.107224 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/1.log" Oct 31 16:02:33.107294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.107254 4880 generic.go:356] "Generic (PLEG): container finished" podID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerID="ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732" exitCode=255 Oct 31 16:02:33.122119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.122082 4880 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="8ab2e984d27532ccecdd52bf8c66804c13a01678ad320e5d7d517bebb63c1413" exitCode=0 Oct 31 16:02:33.122119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.122113 4880 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="3262a0f36fb2867ed33c589c28a1862b244992ded17c11ce8df1fef9ebf61e0e" exitCode=0 Oct 31 16:02:33.122119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.122119 4880 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="5840407302e9d1e0b1b647c250bd99a99a812ff4277fe6b20f36dde6371a7c80" exitCode=0 Oct 31 16:02:33.122119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.122124 4880 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="2497e714c76f977665ec9b4b4cab0d7bf8b540710d962b43da5b5c7a9dc14c07" exitCode=0 Oct 31 16:02:33.122119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.122129 4880 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="7664a9a552a773662349992f61d1f71deb09fffd34dfd35f381ec1264a19440b" exitCode=0 Oct 31 16:02:33.122119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.122147 4880 generic.go:356] "Generic (PLEG): container finished" podID="37a8427f-f59a-4c8e-8c1a-67791b68f4a5" containerID="0202b4be9b346f2a69a3ecc7aa86ffc678ff9c071bc05a5ef056eff6390ee131" exitCode=0 Oct 31 16:02:33.126047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.126018 4880 generic.go:356] "Generic (PLEG): container finished" podID="3a45f772b2761dc4954a13fe159427f7" containerID="a35766bb434ccc7851b7bddb23d31e53cab4abf2e98d5e786e14f1c3191c3978" exitCode=0 Oct 31 16:02:33.138914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.138885 4880 manager.go:324] Recovery completed Oct 31 16:02:33.146014 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.145981 4880 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" Oct 31 16:02:33.158759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.158735 4880 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:02:33.160046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.160025 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:02:33.160146 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.160063 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:02:33.160146 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.160077 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:02:33.162182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162163 4880 cpu_manager.go:226] "Starting" policy="none" Oct 31 16:02:33.162182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162178 4880 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 16:02:33.162328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162198 4880 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 16:02:33.162328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162169 4880 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:02:33.162328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162323 4880 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 16:02:33.162459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162333 4880 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 16:02:33.162459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162353 4880 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 16:02:33.162459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162361 4880 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 16:02:33.162459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162368 4880 policy_none.go:50] "Start" Oct 31 16:02:33.162459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162393 4880 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 16:02:33.162459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162403 4880 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 16:02:33.162685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162568 4880 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 16:02:33.162685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162579 4880 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 16:02:33.162685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.162588 4880 policy_none.go:44] "Start" Oct 31 16:02:33.163163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.163131 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:02:33.163241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.163180 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:02:33.163241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.163195 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:02:33.167044 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.167015 4880 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 16:02:33.169610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.169597 4880 manager.go:341] "Starting Device Plugin manager" Oct 31 16:02:33.169703 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.169623 4880 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 16:02:33.169703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.169636 4880 server.go:87] "Starting device plugin registration server" Oct 31 16:02:33.169915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.169902 4880 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 16:02:33.169971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.169914 4880 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 16:02:33.170016 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.169997 4880 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 16:02:33.170117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.170088 4880 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 16:02:33.170188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.170128 4880 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 16:02:33.170188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.170155 4880 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 16:02:33.170289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.170209 4880 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T16:02:33.170115588Z","UUID":"9ba3cf86-7978-44c3-9b3e-55b8611089f1","Handler":null,"Name":"","Endpoint":""} Oct 31 16:02:33.172378 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.172337 4880 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 16:02:33.172487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.172351 4880 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 16:02:33.172546 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.172491 4880 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" Oct 31 16:02:33.172546 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.172518 4880 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 16:02:33.181221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.181192 4880 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:02:33.182513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.182463 4880 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal"] Oct 31 16:02:33.182621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.182598 4880 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:02:33.183488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.183471 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:02:33.183534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.183506 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:02:33.183534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.183519 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:02:33.183837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.183826 4880 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:02:33.183978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.183964 4880 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:02:33.184586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.184572 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:02:33.184665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.184603 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:02:33.184665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.184618 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:02:33.184665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.184572 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:02:33.184817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.184682 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:02:33.184817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.184694 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:02:33.184817 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.184797 4880 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.185087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185074 4880 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:02:33.185163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185099 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"3a45f772b2761dc4954a13fe159427f7","Type":"ContainerStarted","Data":"8634b55a03d4b691ac186ca506faac98236b737b8be67ed3c93e87c2645625eb"} Oct 31 16:02:33.185216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185164 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"3a45f772b2761dc4954a13fe159427f7","Type":"ContainerDied","Data":"a35766bb434ccc7851b7bddb23d31e53cab4abf2e98d5e786e14f1c3191c3978"} Oct 31 16:02:33.185216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185180 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"3a45f772b2761dc4954a13fe159427f7","Type":"ContainerStarted","Data":"05e06b5a017a76f6498bee3eddc63fbf325a4f10e00b5fabc3adf6ed0f135b7a"} Oct 31 16:02:33.185216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185206 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"10be822f2f376ae4d26eb7bb3d2aeb64","Type":"ContainerStarted","Data":"0a54cf93dfe48cc6e8eefc6b6d77027f317d5800525253d4ebd7ec45d7c9f0cc"} Oct 31 16:02:33.185311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185218 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" event={"ID":"10be822f2f376ae4d26eb7bb3d2aeb64","Type":"ContainerStarted","Data":"9500ec4905fa31fe5e237f6394111793e21eb7dd70ce3fe09cfbfd03b4215d00"} Oct 31 16:02:33.185840 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185828 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:02:33.185887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185851 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:02:33.185887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.185861 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:02:33.186001 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.185988 4880 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-127.us-west-2.compute.internal\" not found" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.225726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.225693 4880 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:02:33.227243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.227225 4880 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:02:33.246853 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.246821 4880 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.271006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.270976 4880 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 16:02:33.272398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.272377 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 16:02:33.272481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.272412 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 16:02:33.272481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.272424 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 16:02:33.272481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.272455 4880 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.273925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.273905 4880 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:02:33.274251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.274220 4880 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 16:02:33.302545 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.302518 4880 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 16:02:33.302649 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.302614 4880 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.302649 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.302630 4880 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.373901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.373825 4880 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.374033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.373908 4880 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.374384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.374360 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10be822f2f376ae4d26eb7bb3d2aeb64-config\") pod \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"10be822f2f376ae4d26eb7bb3d2aeb64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.374483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.374393 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.374483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.374420 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.374571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.374494 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeNotReady" Oct 31 16:02:33.374571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.374520 4880 setters.go:546] "Node became not ready" node="ip-10-0-142-127.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T16:02:33Z","lastTransitionTime":"2026-10-31T16:02:33Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 16:02:33.376656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.376636 4880 nodeinfomanager.go:471] existing CSINode "ip-10-0-142-127.us-west-2.compute.internal" is owned by different node (oldNodeID="f253db34-638c-4510-9de1-db414e165bcc", newNodeID=""), cleaning up... Oct 31 16:02:33.404345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.404315 4880 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 16:02:33.404431 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:33.404372 4880 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.475194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.475158 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10be822f2f376ae4d26eb7bb3d2aeb64-config\") pod \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"10be822f2f376ae4d26eb7bb3d2aeb64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.475194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.475195 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.475376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.475203 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10be822f2f376ae4d26eb7bb3d2aeb64-config\") pod \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"10be822f2f376ae4d26eb7bb3d2aeb64\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.475376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.475216 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.475376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.475247 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.475376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.475277 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a45f772b2761dc4954a13fe159427f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" (UID: \"3a45f772b2761dc4954a13fe159427f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:33.637001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.636976 4880 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-127.us-west-2.compute.internal" event="NodeReady" Oct 31 16:02:33.637164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:33.637061 4880 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 16:02:34.031248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.031159 4880 apiserver.go:52] "Watching apiserver" Oct 31 16:02:34.135965 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.135938 4880 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:34.136128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.136086 4880 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:34.174237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.174205 4880 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 16:02:34.176744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.176705 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-s7jw9","openshift-network-diagnostics/network-check-target-kddfn","kube-system/kas-connection-checker-685895c7c9-9cdsm","openshift-console-operator/console-operator-7cd56b8dbc-bthm6","openshift-insights/insights-operator-7c4986d449-9zgx8","openshift-insights/insights-runtime-extractor-vs98d","openshift-multus/network-metrics-daemon-mmp6g","openshift-network-operator/iptables-alerter-9wns7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-kns59","kube-system/kas-connection-checker-685895c7c9-x72jc","kube-system/global-pull-secret-syncer-mvztk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8","openshift-multus/multus-additional-cni-plugins-hwvf5","openshift-dns/dns-default-b2gc9","openshift-ovn-kubernetes/ovnkube-node-m6xpv","kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf","openshift-cluster-node-tuning-operator/tuned-blwvk","openshift-network-console/networking-console-plugin-86c478bb88-m76mb","openshift-image-registry/node-ca-nmnvx","openshift-ingress-canary/ingress-canary-sgkl6","openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs","kube-system/konnectivity-agent-4wccg","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw","openshift-ingress/router-default-746fc4dd59-qfxr9","openshift-image-registry/image-registry-7b9f8d5c7-c4czr","openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq","kube-system/kas-connection-checker-685895c7c9-69955","openshift-multus/multus-6wd5x","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl","openshift-image-registry/image-registry-98675d9cb-646m2"] Oct 31 16:02:34.178539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.178515 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.178824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.178809 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.179202 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.179184 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.180078 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.180064 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.181592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.181576 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:34.184115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.184096 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:34.187969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.187950 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.188514 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.188500 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.189548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.189530 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.193100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.193078 4880 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 16:02:34.193208 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.193170 4880 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:34.193208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.193197 4880 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 16:02:34.193326 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.193309 4880 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-127.us-west-2.compute.internal" Oct 31 16:02:34.227191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.227163 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.227555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.227539 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.228166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228124 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.228166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228149 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228149 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228189 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228210 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228213 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228214 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228306 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228330 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228333 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228359 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 16:02:34.228360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228373 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228392 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228429 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228453 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228528 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228540 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228600 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228616 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-85rcs\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228670 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228735 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nfvld\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228736 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228741 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lhk9l\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228741 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228769 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 16:02:34.228851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228800 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 16:02:34.229292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.228949 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 16:02:34.229690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.229671 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.229784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.229692 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.229784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.229742 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.229845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.229806 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.229845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.229829 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 16:02:34.230211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.230197 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 16:02:34.230248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.230234 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 16:02:34.230248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.230236 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.230318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.230260 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.230318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.230265 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 16:02:34.230845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.230823 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.231053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.231028 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dstd9\"" Oct 31 16:02:34.231130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.231061 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.231130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.231063 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.248280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.248259 4880 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 16:02:34.269079 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.269054 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.274844 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.274823 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.274844 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.274838 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.275047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.274925 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 16:02:34.275047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.274959 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 16:02:34.275161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275078 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s8pkl\"" Oct 31 16:02:34.275161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275103 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 16:02:34.275161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275122 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jqwtc\"" Oct 31 16:02:34.275260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275218 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.275319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275304 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.275457 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275445 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wg2f6\"" Oct 31 16:02:34.275518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275507 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jwpwk\"" Oct 31 16:02:34.275571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275555 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 16:02:34.275615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275579 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 16:02:34.275661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.275639 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vnjtb\"" Oct 31 16:02:34.277052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277032 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.277164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277061 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 16:02:34.277164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277082 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.277267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277228 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 16:02:34.277315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277265 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 16:02:34.277315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277273 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.277315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277277 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 16:02:34.277315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277288 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.277315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277313 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277317 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277325 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277333 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277349 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277374 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277392 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277439 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277449 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277453 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277483 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 16:02:34.277513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277516 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277541 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277544 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277544 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mwvsn\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277568 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f9xg5\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277597 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnhm9\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277613 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277632 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fd4\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277659 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-msflc\"" Oct 31 16:02:34.277936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277790 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgxcq\"" Oct 31 16:02:34.278272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.277957 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 16:02:34.278272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278042 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 16:02:34.278272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278118 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d4rvf\"" Oct 31 16:02:34.278272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278171 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.278272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278238 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9wrg2\"" Oct 31 16:02:34.278272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278245 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4fqcg\"" Oct 31 16:02:34.278434 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278318 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tcbkl\"" Oct 31 16:02:34.278434 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278337 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zwdm2\"" Oct 31 16:02:34.278434 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278362 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 16:02:34.278529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278462 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7g99\"" Oct 31 16:02:34.278529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278477 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.278529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278524 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 16:02:34.278662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278599 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 16:02:34.278662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278602 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-service-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.278662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278630 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce582737-4c20-4d14-9199-ef952d229b5a-ovn-node-metrics-cert\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.278780 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278669 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 16:02:34.278780 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278711 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tp4vw\"" Oct 31 16:02:34.278780 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278773 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-conf-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.278914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278810 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.278914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278820 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce582737-4c20-4d14-9199-ef952d229b5a-ovn-node-metrics-cert\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.278914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278840 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:34.278914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278821 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-service-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.278914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278890 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn85\" (UniqueName: \"kubernetes.io/projected/ed820d3a-f16c-4bf0-a30e-85ad2732ab58-kube-api-access-rgn85\") pod \"kas-connection-checker-685895c7c9-9cdsm\" (UID: \"ed820d3a-f16c-4bf0-a30e-85ad2732ab58\") " pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" Oct 31 16:02:34.279103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.278939 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-kubernetes\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.279103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279000 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-systemd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.279103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279049 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.279103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279081 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.279260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279116 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.279260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279199 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hm7f\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.279534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279513 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-tuned\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.279605 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279565 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sd5\" (UniqueName: \"kubernetes.io/projected/65899ffa-6ce0-4db1-8759-b17f99943f51-kube-api-access-j9sd5\") pod \"network-check-source-6f78d59f99-hlsdq\" (UID: \"65899ffa-6ce0-4db1-8759-b17f99943f51\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" Oct 31 16:02:34.279650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279606 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgj6\" (UniqueName: \"kubernetes.io/projected/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-kube-api-access-6pgj6\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:02:34.279650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279638 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cni-binary-copy\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.279739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279665 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468a8f95-953e-403c-a2d7-931d530da8ee-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:34.279739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279700 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8vd\" (UniqueName: \"kubernetes.io/projected/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-kube-api-access-mk8vd\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.279827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279780 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-tuned\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.279872 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279858 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.279914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279861 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4e8bad-6ee9-4270-a117-70210189abe1-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.279959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279938 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.280005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.279973 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-multus\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.280051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280004 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrm9r\" (UniqueName: \"kubernetes.io/projected/55a6396f-e3a0-42ec-88fa-c4d23384e807-kube-api-access-wrm9r\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:34.280051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280040 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs6c\" (UniqueName: \"kubernetes.io/projected/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-kube-api-access-cqs6c\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.280156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280069 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-config-volume\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.280156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280100 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.280156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280118 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cni-binary-copy\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.280156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280131 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-var-lib-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.280362 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280339 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/468a8f95-953e-403c-a2d7-931d530da8ee-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:34.280543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280508 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-config-volume\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.280663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280584 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-script-lib\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.280663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280638 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5p2\" (UniqueName: \"kubernetes.io/projected/468a8f95-953e-403c-a2d7-931d530da8ee-kube-api-access-2g5p2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:34.280810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280795 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-var-lib-kubelet\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.280884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280840 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-tmp\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.280884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280877 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.281128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280932 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.281128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.280972 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbvr\" (UniqueName: \"kubernetes.io/projected/eb9982b4-6251-4984-8493-e884e03f13ae-kube-api-access-9nbvr\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:02:34.281128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281020 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhk2\" (UniqueName: \"kubernetes.io/projected/fb4e8bad-6ee9-4270-a117-70210189abe1-kube-api-access-xhhk2\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.281128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281045 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-script-lib\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.281391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281205 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.281391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281259 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.281391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281354 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.281525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281391 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.281525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281406 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.281525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281473 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.281652 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281550 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-tmp\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.281835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.281818 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.282501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282480 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.282591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282540 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-config\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.282591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282568 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-hostroot\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.282739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282631 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-daemon-config\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.282739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282663 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wzd\" (UniqueName: \"kubernetes.io/projected/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-kube-api-access-78wzd\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.282739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282696 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:34.282739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282729 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9982b4-6251-4984-8493-e884e03f13ae-host-slash\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:02:34.282961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282803 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-systemd\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.282961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282833 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-netns\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.282961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282869 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-dbus\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.282961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282901 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5g9\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.282961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282925 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.282961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282959 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vm2\" (UniqueName: \"kubernetes.io/projected/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-kube-api-access-29vm2\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.283284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.282989 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e06bd6-7368-4e48-8732-782df89e89dd-cert\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:34.283284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283020 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-host\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:02:34.283284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283121 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.283284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283173 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-lib-modules\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.283284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283206 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-env-overrides\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283293 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-bin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283358 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43a99cec-f100-473d-b831-f8052ae72bf0-konnectivity-ca\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283384 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283417 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzzb\" (UniqueName: \"kubernetes.io/projected/e9cad0b0-2b17-48f6-9c14-12a419791b1f-kube-api-access-bdzzb\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283450 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-netd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283483 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-kubelet-config\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283514 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283543 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468a8f95-953e-403c-a2d7-931d530da8ee-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283575 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-conf\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.283629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283607 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-systemd-units\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.284112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283645 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-device-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.284112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283676 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cnibin\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.284112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.283704 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-serviceca\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:02:34.284112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284094 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-ovnkube-config\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.284300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284131 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.284300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284201 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-k8s-cni-cncf-io\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.284300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284280 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e06bd6-7368-4e48-8732-782df89e89dd-cert\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:34.284300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284292 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43a99cec-f100-473d-b831-f8052ae72bf0-konnectivity-ca\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:02:34.284481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284329 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-multus-certs\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.284481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284353 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.284481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284415 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.284481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284462 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.284692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284500 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-daemon-config\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.284692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284524 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-slash\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.284692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284559 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-node-log\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.284692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284588 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce582737-4c20-4d14-9199-ef952d229b5a-env-overrides\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.284692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284620 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-bin\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.284692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284657 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b75f93-390b-4813-9264-a9b9709fc330-crio-socket\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.284692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284676 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-serviceca\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:02:34.285010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284699 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.285010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284723 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.285010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284812 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.285010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284857 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.285010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284907 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-tmp-dir\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.285010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284964 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpgjb\" (UniqueName: \"kubernetes.io/projected/45d11ee0-d36c-4f4a-915c-fcedef344e68-kube-api-access-fpgjb\") pod \"kas-connection-checker-685895c7c9-69955\" (UID: \"45d11ee0-d36c-4f4a-915c-fcedef344e68\") " pod="kube-system/kas-connection-checker-685895c7c9-69955" Oct 31 16:02:34.285010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.284989 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:34.285333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285023 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/468a8f95-953e-403c-a2d7-931d530da8ee-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:34.285333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285076 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.285333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285075 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.285333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285115 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.285333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285168 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.285333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285239 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9ll\" (UniqueName: \"kubernetes.io/projected/95a4cede-89fb-421e-be64-69f3ea494179-kube-api-access-5k9ll\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:02:34.285333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285306 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-tmp-dir\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.285515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285352 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/44c3511b-f377-4032-a196-2490d71549ae-kube-api-access-plhvw\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.285515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285429 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.285833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285814 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-tmp\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.285881 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285864 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-sys-fs\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.285930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285896 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96z7w\" (UniqueName: \"kubernetes.io/projected/efec2af7-bad8-43d3-8fd8-45071136bdcf-kube-api-access-96z7w\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.285994 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.285979 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95a4cede-89fb-421e-be64-69f3ea494179-hosts-file\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:02:34.286040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286014 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e3677f-537d-4cd9-8b72-3486e64246b6-serving-cert\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:34.286093 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286057 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.286161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286083 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910df55b-a9db-4aa6-acc1-3721c868beb8-tmp\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.286161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286107 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-stats-auth\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.286268 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286178 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/44e3677f-537d-4cd9-8b72-3486e64246b6-serving-cert\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:34.286268 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286179 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b75f93-390b-4813-9264-a9b9709fc330-data-volume\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.286379 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286293 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.286379 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286296 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b75f93-390b-4813-9264-a9b9709fc330-data-volume\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.286379 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286325 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-etc-kubernetes\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.286379 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286354 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-snapshots\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.286533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286384 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.286533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286391 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-stats-auth\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.286533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286432 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fb4e8bad-6ee9-4270-a117-70210189abe1-snapshots\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.286533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286435 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jbs\" (UniqueName: \"kubernetes.io/projected/68b75f93-390b-4813-9264-a9b9709fc330-kube-api-access-q6jbs\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.286533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286448 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.286533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286492 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysconfig\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.286789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286599 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.286789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286625 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43a99cec-f100-473d-b831-f8052ae72bf0-agent-certs\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:02:34.286789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286701 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.286789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286736 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.286789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286755 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.287054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286827 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43a99cec-f100-473d-b831-f8052ae72bf0-agent-certs\") pod \"konnectivity-agent-4wccg\" (UID: \"43a99cec-f100-473d-b831-f8052ae72bf0\") " pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:02:34.287054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286921 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-socket-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.287054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286947 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.287054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286975 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a4cede-89fb-421e-be64-69f3ea494179-tmp-dir\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:02:34.287054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.286993 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a66684b-1ce2-4892-b466-d95589fcf351-original-pull-secret\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.287054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.287053 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wsd9\" (UniqueName: \"kubernetes.io/projected/44e3677f-537d-4cd9-8b72-3486e64246b6-kube-api-access-4wsd9\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:34.287341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.287075 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95a4cede-89fb-421e-be64-69f3ea494179-tmp-dir\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:02:34.287341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.287114 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6st7s\" (UniqueName: \"kubernetes.io/projected/b3d75aa1-14ee-4dd6-8216-440abcd707fa-kube-api-access-6st7s\") pod \"kas-connection-checker-685895c7c9-x72jc\" (UID: \"b3d75aa1-14ee-4dd6-8216-440abcd707fa\") " pod="kube-system/kas-connection-checker-685895c7c9-x72jc" Oct 31 16:02:34.287341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.287242 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb926\" (UniqueName: \"kubernetes.io/projected/be3d02e8-cb60-4762-b79f-e29afc5b53ac-kube-api-access-qb926\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.287341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.287272 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpvx\" (UniqueName: \"kubernetes.io/projected/910df55b-a9db-4aa6-acc1-3721c868beb8-kube-api-access-8tpvx\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.287341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.287303 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-ovn\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.287537 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.287381 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.288237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288219 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-kubelet\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.288321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288254 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb4e8bad-6ee9-4270-a117-70210189abe1-serving-cert\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.288321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288280 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e3677f-537d-4cd9-8b72-3486e64246b6-config\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:34.288419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288326 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.288419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288362 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.288419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288392 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb4e8bad-6ee9-4270-a117-70210189abe1-serving-cert\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.288574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288448 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqp5\" (UniqueName: \"kubernetes.io/projected/f5e06bd6-7368-4e48-8732-782df89e89dd-kube-api-access-5mqp5\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:34.288574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288482 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-sys\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.288574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288508 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-etc-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.288574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288516 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.288574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288453 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/44e3677f-537d-4cd9-8b72-3486e64246b6-config\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:34.288574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288535 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-netns\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.288574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288557 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.288866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288577 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.288866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288617 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86bf6\" (UniqueName: \"kubernetes.io/projected/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222-kube-api-access-86bf6\") pod \"volume-data-source-validator-58c9f5f46d-xx5j8\" (UID: \"7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" Oct 31 16:02:34.288866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288644 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-registration-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.288866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288698 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.288866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288718 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-default-certificate\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.288866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288739 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.288866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288844 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db4c92c3-fe37-4fbf-be54-a0196309c59c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288941 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288944 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-run\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.288984 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-system-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289003 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9982b4-6251-4984-8493-e884e03f13ae-iptables-alerter-script\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289036 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-default-certificate\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289051 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-trusted-ca\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289038 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-config\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289111 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db4c92c3-fe37-4fbf-be54-a0196309c59c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289107 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9cad0b0-2b17-48f6-9c14-12a419791b1f-serving-cert\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289169 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-modprobe-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289193 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45vjp\" (UniqueName: \"kubernetes.io/projected/ce582737-4c20-4d14-9199-ef952d229b5a-kube-api-access-45vjp\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289212 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9cad0b0-2b17-48f6-9c14-12a419791b1f-config\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.289229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289221 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-socket-dir-parent\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.289598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289257 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.289598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289265 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9cad0b0-2b17-48f6-9c14-12a419791b1f-serving-cert\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.289598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289285 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-host\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.289598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289296 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb9982b4-6251-4984-8493-e884e03f13ae-iptables-alerter-script\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:02:34.289598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289316 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-kubelet\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.289598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289342 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-log-socket\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.289598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.289373 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cnibin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.331404 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.331368 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hm7f\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.331781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.331764 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn85\" (UniqueName: \"kubernetes.io/projected/ed820d3a-f16c-4bf0-a30e-85ad2732ab58-kube-api-access-rgn85\") pod \"kas-connection-checker-685895c7c9-9cdsm\" (UID: \"ed820d3a-f16c-4bf0-a30e-85ad2732ab58\") " pod="kube-system/kas-connection-checker-685895c7c9-9cdsm" Oct 31 16:02:34.333281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.332960 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.333281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333037 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqs6c\" (UniqueName: \"kubernetes.io/projected/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-kube-api-access-cqs6c\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.333281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333044 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgj6\" (UniqueName: \"kubernetes.io/projected/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-kube-api-access-6pgj6\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:02:34.333281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333085 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbvr\" (UniqueName: \"kubernetes.io/projected/eb9982b4-6251-4984-8493-e884e03f13ae-kube-api-access-9nbvr\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:02:34.333281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333235 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5p2\" (UniqueName: \"kubernetes.io/projected/468a8f95-953e-403c-a2d7-931d530da8ee-kube-api-access-2g5p2\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl\" (UID: \"468a8f95-953e-403c-a2d7-931d530da8ee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" Oct 31 16:02:34.333281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333277 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8vd\" (UniqueName: \"kubernetes.io/projected/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-kube-api-access-mk8vd\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.333541 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333292 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrm9r\" (UniqueName: \"kubernetes.io/projected/55a6396f-e3a0-42ec-88fa-c4d23384e807-kube-api-access-wrm9r\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:34.333541 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333524 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sd5\" (UniqueName: \"kubernetes.io/projected/65899ffa-6ce0-4db1-8759-b17f99943f51-kube-api-access-j9sd5\") pod \"network-check-source-6f78d59f99-hlsdq\" (UID: \"65899ffa-6ce0-4db1-8759-b17f99943f51\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-hlsdq" Oct 31 16:02:34.333783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333768 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.333910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.333895 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhk2\" (UniqueName: \"kubernetes.io/projected/fb4e8bad-6ee9-4270-a117-70210189abe1-kube-api-access-xhhk2\") pod \"insights-operator-7c4986d449-9zgx8\" (UID: \"fb4e8bad-6ee9-4270-a117-70210189abe1\") " pod="openshift-insights/insights-operator-7c4986d449-9zgx8" Oct 31 16:02:34.335001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.334975 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vm2\" (UniqueName: \"kubernetes.io/projected/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-kube-api-access-29vm2\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.335314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.335299 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5g9\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.335675 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.335660 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbg9n\" (UniqueName: \"kubernetes.io/projected/ec9fc385-70b6-442b-b076-0102e41c5fbd-kube-api-access-qbg9n\") pod \"network-check-target-kddfn\" (UID: \"ec9fc385-70b6-442b-b076-0102e41c5fbd\") " pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:34.336730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.336706 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9ll\" (UniqueName: \"kubernetes.io/projected/95a4cede-89fb-421e-be64-69f3ea494179-kube-api-access-5k9ll\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:02:34.336965 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.336939 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wzd\" (UniqueName: \"kubernetes.io/projected/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-kube-api-access-78wzd\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.337243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.337227 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jbs\" (UniqueName: \"kubernetes.io/projected/68b75f93-390b-4813-9264-a9b9709fc330-kube-api-access-q6jbs\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.337299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.337235 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpgjb\" (UniqueName: \"kubernetes.io/projected/45d11ee0-d36c-4f4a-915c-fcedef344e68-kube-api-access-fpgjb\") pod \"kas-connection-checker-685895c7c9-69955\" (UID: \"45d11ee0-d36c-4f4a-915c-fcedef344e68\") " pod="kube-system/kas-connection-checker-685895c7c9-69955" Oct 31 16:02:34.337299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.337232 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzzb\" (UniqueName: \"kubernetes.io/projected/e9cad0b0-2b17-48f6-9c14-12a419791b1f-kube-api-access-bdzzb\") pod \"console-operator-7cd56b8dbc-bthm6\" (UID: \"e9cad0b0-2b17-48f6-9c14-12a419791b1f\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:34.337761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.337746 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wsd9\" (UniqueName: \"kubernetes.io/projected/44e3677f-537d-4cd9-8b72-3486e64246b6-kube-api-access-4wsd9\") pod \"service-ca-operator-544d749c8c-6srcs\" (UID: \"44e3677f-537d-4cd9-8b72-3486e64246b6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" Oct 31 16:02:34.337962 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.337947 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhvw\" (UniqueName: \"kubernetes.io/projected/44c3511b-f377-4032-a196-2490d71549ae-kube-api-access-plhvw\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.337998 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.337945 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6st7s\" (UniqueName: \"kubernetes.io/projected/b3d75aa1-14ee-4dd6-8216-440abcd707fa-kube-api-access-6st7s\") pod \"kas-connection-checker-685895c7c9-x72jc\" (UID: \"b3d75aa1-14ee-4dd6-8216-440abcd707fa\") " pod="kube-system/kas-connection-checker-685895c7c9-x72jc" Oct 31 16:02:34.338234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.338216 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96z7w\" (UniqueName: \"kubernetes.io/projected/efec2af7-bad8-43d3-8fd8-45071136bdcf-kube-api-access-96z7w\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.338637 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.338623 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpvx\" (UniqueName: \"kubernetes.io/projected/910df55b-a9db-4aa6-acc1-3721c868beb8-kube-api-access-8tpvx\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.338979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.338963 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb926\" (UniqueName: \"kubernetes.io/projected/be3d02e8-cb60-4762-b79f-e29afc5b53ac-kube-api-access-qb926\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.339472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.339453 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqp5\" (UniqueName: \"kubernetes.io/projected/f5e06bd6-7368-4e48-8732-782df89e89dd-kube-api-access-5mqp5\") pod \"ingress-canary-sgkl6\" (UID: \"f5e06bd6-7368-4e48-8732-782df89e89dd\") " pod="openshift-ingress-canary/ingress-canary-sgkl6" Oct 31 16:02:34.339713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.339696 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45vjp\" (UniqueName: \"kubernetes.io/projected/ce582737-4c20-4d14-9199-ef952d229b5a-kube-api-access-45vjp\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.340256 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.340240 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86bf6\" (UniqueName: \"kubernetes.io/projected/7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222-kube-api-access-86bf6\") pod \"volume-data-source-validator-58c9f5f46d-xx5j8\" (UID: \"7d63f3bd-5f89-4a1d-9ee6-9e0bcb099222\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xx5j8" Oct 31 16:02:34.346683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.346661 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:34.390100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390068 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-host\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:02:34.390100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390104 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-lib-modules\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390121 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-bin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390153 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-netd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390176 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-kubelet-config\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390198 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb6cc68-a2af-438f-83be-9f1e9afc57b6-host\") pod \"node-ca-nmnvx\" (UID: \"9cb6cc68-a2af-438f-83be-9f1e9afc57b6\") " pod="openshift-image-registry/node-ca-nmnvx" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390208 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-bin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390231 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-kubelet-config\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390229 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-netd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390273 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-lib-modules\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.390299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390278 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390307 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-conf\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390324 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-systemd-units\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390342 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-device-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390363 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cnibin\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390383 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-systemd-units\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390386 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-k8s-cni-cncf-io\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390407 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-device-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390415 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-multus-certs\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390433 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-cnibin\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390447 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-multus-certs\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390453 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390418 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-k8s-cni-cncf-io\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.390476 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390478 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-conf\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.390487 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390487 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390485 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390513 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-slash\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390532 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-node-log\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390546 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-bin\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390564 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-slash\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390577 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-cni-bin\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390587 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-node-log\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.390570 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.890552214 +0000 UTC m=+1.925847021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390593 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysctl-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390632 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b75f93-390b-4813-9264-a9b9709fc330-crio-socket\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390652 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390672 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-sys-fs\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.390682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390688 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95a4cede-89fb-421e-be64-69f3ea494179-hosts-file\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390706 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390731 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-etc-kubernetes\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390739 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-sys-fs\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390752 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68b75f93-390b-4813-9264-a9b9709fc330-crio-socket\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390760 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysconfig\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390764 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95a4cede-89fb-421e-be64-69f3ea494179-hosts-file\") pod \"node-resolver-s7jw9\" (UID: \"95a4cede-89fb-421e-be64-69f3ea494179\") " pod="openshift-dns/node-resolver-s7jw9" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390779 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-etc-kubernetes\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.390788 4880 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390798 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-sysconfig\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390797 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390798 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.390821 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.890812389 +0000 UTC m=+1.926107196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390826 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390859 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390883 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-socket-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390910 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-ovn\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390933 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390948 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-socket-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390961 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-kubelet\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390965 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-ovn\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390983 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.390987 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-sys\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391007 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-etc-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391009 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-kubelet\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391031 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-sys\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391029 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-netns\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391050 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-etc-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391061 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-run-netns\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391064 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391088 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391098 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-registration-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391128 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-run\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391165 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/44c3511b-f377-4032-a196-2490d71549ae-registration-dir\") pod \"aws-ebs-csi-driver-node-pp9bf\" (UID: \"44c3511b-f377-4032-a196-2490d71549ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pp9bf" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391168 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-system-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391198 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-modprobe-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391220 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-run\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391223 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-socket-dir-parent\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391258 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-system-cni-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391267 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-socket-dir-parent\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391284 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-modprobe-d\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391291 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391260 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391329 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-host\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391351 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-kubelet\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391376 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-log-socket\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391396 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-kubelet\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391399 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cnibin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391406 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68b75f93-390b-4813-9264-a9b9709fc330-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391422 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-log-socket\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391425 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-conf-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391445 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-multus-conf-dir\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391379 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-host\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391452 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391477 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-kubernetes\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391456 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-cnibin\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391505 4880 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391500 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-systemd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391536 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391546 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-run-systemd\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391542 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-kubernetes\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391566 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.891534785 +0000 UTC m=+1.926829568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : secret "metrics-daemon-secret" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391584 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37a8427f-f59a-4c8e-8c1a-67791b68f4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwvf5\" (UID: \"37a8427f-f59a-4c8e-8c1a-67791b68f4a5\") " pod="openshift-multus/multus-additional-cni-plugins-hwvf5" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391602 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391620 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391653 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391680 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391692 4880 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391704 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-multus\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391733 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391739 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391754 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391757 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-host-var-lib-cni-multus\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391776 4880 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391777 4880 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391741 4880 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391744 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.89173017 +0000 UTC m=+1.927024963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391858 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.891850213 +0000 UTC m=+1.927144996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391872 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.891866295 +0000 UTC m=+1.927161079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : secret "insights-runtime-extractor-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391883 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.891877859 +0000 UTC m=+1.927172642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.391895 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.891889529 +0000 UTC m=+1.927184312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:34.391855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391911 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-var-lib-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391928 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-var-lib-kubelet\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391950 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391965 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.391980 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-hostroot\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392000 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9982b4-6251-4984-8493-e884e03f13ae-host-slash\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392015 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392019 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-var-lib-kubelet\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.392039 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.89202716 +0000 UTC m=+1.927321960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392038 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-var-lib-openvswitch\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392048 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6483e8c0-5b7a-46c6-b270-1f741dd0e8ad-hostroot\") pod \"multus-6wd5x\" (UID: \"6483e8c0-5b7a-46c6-b270-1f741dd0e8ad\") " pod="openshift-multus/multus-6wd5x" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392039 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb9982b4-6251-4984-8493-e884e03f13ae-host-slash\") pod \"iptables-alerter-9wns7\" (UID: \"eb9982b4-6251-4984-8493-e884e03f13ae\") " pod="openshift-network-operator/iptables-alerter-9wns7" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392062 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-systemd\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392084 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-netns\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392118 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-dbus\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392129 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce582737-4c20-4d14-9199-ef952d229b5a-host-run-netns\") pod \"ovnkube-node-m6xpv\" (UID: \"ce582737-4c20-4d14-9199-ef952d229b5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392150 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910df55b-a9db-4aa6-acc1-3721c868beb8-etc-systemd\") pod \"tuned-blwvk\" (UID: \"910df55b-a9db-4aa6-acc1-3721c868beb8\") " pod="openshift-cluster-node-tuning-operator/tuned-blwvk" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392162 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.392223 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a66684b-1ce2-4892-b466-d95589fcf351-dbus\") pod \"global-pull-secret-syncer-mvztk\" (UID: \"5a66684b-1ce2-4892-b466-d95589fcf351\") " pod="kube-system/global-pull-secret-syncer-mvztk" Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.392238 4880 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:34.394121 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.392278 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:34.892269304 +0000 UTC m=+1.927564164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:34.480185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.480155 4880 scope.go:122] "RemoveContainer" containerID="ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732" Oct 31 16:02:34.896363 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896333 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:34.896522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896372 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:34.896522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896390 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:34.896522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896411 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:34.896522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896430 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:34.896522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896453 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:34.896522 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896497 4880 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896525 4880 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896558 4880 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896575 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896554595 +0000 UTC m=+2.931849398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896593 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896584484 +0000 UTC m=+2.931879267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : secret "insights-runtime-extractor-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896497 4880 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896608 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896600368 +0000 UTC m=+2.931895155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896626 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896500 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896644 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896638944 +0000 UTC m=+2.931933728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : secret "metrics-daemon-secret" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896651 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896678 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896667177 +0000 UTC m=+2.931961964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896675 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896509 4880 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896695 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.89668519 +0000 UTC m=+2.931979999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896715 4880 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896725 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896715997 +0000 UTC m=+2.932010779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896738 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896732292 +0000 UTC m=+2.932027133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896768 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:34.896797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:34.896799 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:34.897468 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896874 4880 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:34.897468 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896902 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896896274 +0000 UTC m=+2.932191057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:34.897468 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896910 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:34.897468 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.896929 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:34.897468 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:34.897013 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:35.896997593 +0000 UTC m=+2.932292397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:35.140798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.140767 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/2.log" Oct 31 16:02:35.141270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.141244 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/1.log" Oct 31 16:02:35.141311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.141272 4880 generic.go:356] "Generic (PLEG): container finished" podID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerID="d46a25067123d6d611de659ec0444e2ec038629265b823b3dd7a1fc4ea5b5a3f" exitCode=255 Oct 31 16:02:35.141370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.141350 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" event={"ID":"e9cad0b0-2b17-48f6-9c14-12a419791b1f","Type":"ContainerDied","Data":"d46a25067123d6d611de659ec0444e2ec038629265b823b3dd7a1fc4ea5b5a3f"} Oct 31 16:02:35.141425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.141413 4880 scope.go:122] "RemoveContainer" containerID="ef802c8867293b0dea98d7405114e13108fef138948718502cd6617e477aa732" Oct 31 16:02:35.141901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.141883 4880 scope.go:122] "RemoveContainer" containerID="d46a25067123d6d611de659ec0444e2ec038629265b823b3dd7a1fc4ea5b5a3f" Oct 31 16:02:35.142055 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.142037 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:35.187311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.187276 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:35.401449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.401418 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:35.553759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.553676 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:35.907424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907384 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:35.907424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907429 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:35.907606 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907523 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:35.907606 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907525 4880 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:35.907606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907553 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:35.907606 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907596 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.907577384 +0000 UTC m=+4.942872188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:35.907606 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907533 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:35.907795 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907631 4880 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:35.907795 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907662 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.907645733 +0000 UTC m=+4.942940536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:35.907795 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907683 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.907676903 +0000 UTC m=+4.942971686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:35.907795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907718 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:35.907795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907741 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:35.907795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907757 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907809 4880 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907826 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907834 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.907827167 +0000 UTC m=+4.943121950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : secret "metrics-daemon-secret" not found Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907837 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907855 4880 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907869 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.907858879 +0000 UTC m=+4.943153684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.907905 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.907897085 +0000 UTC m=+4.943191872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907946 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.907995 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:35.908030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.908029 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.908047 4880 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.908061 4880 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.908071 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.908064334 +0000 UTC m=+4.943359153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:35.908067 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.908083 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.90807681 +0000 UTC m=+4.943371592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.908113 4880 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.908155 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.908148855 +0000 UTC m=+4.943443663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : secret "insights-runtime-extractor-tls" not found Oct 31 16:02:35.908415 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:35.908194 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:37.908180235 +0000 UTC m=+4.943475032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:36.145781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:36.145751 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/2.log" Oct 31 16:02:36.146198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:36.146096 4880 scope.go:122] "RemoveContainer" containerID="d46a25067123d6d611de659ec0444e2ec038629265b823b3dd7a1fc4ea5b5a3f" Oct 31 16:02:36.146317 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:36.146297 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:36.362515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:36.362440 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:37.927815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927773 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927840 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927859 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927874 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927896 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927913 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927936 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.927953 4880 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.927969 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928003 4880 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.928005 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928037 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928016824 +0000 UTC m=+8.963311609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928011 4880 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928049 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928066 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928054955 +0000 UTC m=+8.963349746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : secret "metrics-daemon-secret" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928070 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928048 4880 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928086 4880 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928088 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928073149 +0000 UTC m=+8.963367934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928175 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928161301 +0000 UTC m=+8.963456096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928195 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928183818 +0000 UTC m=+8.963478641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928207 4880 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928213 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928204211 +0000 UTC m=+8.963498997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928229 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928221565 +0000 UTC m=+8.963516397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928246 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls podName:68b75f93-390b-4813-9264-a9b9709fc330 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928237744 +0000 UTC m=+8.963532527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vs98d" (UID: "68b75f93-390b-4813-9264-a9b9709fc330") : secret "insights-runtime-extractor-tls" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928264 4880 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.928274 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:37.928298 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928299 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928288435 +0000 UTC m=+8.963583240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:37.929084 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928329 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:37.929084 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928337 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:37.929084 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.928361 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:41.928352312 +0000 UTC m=+8.963647121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:37.953779 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.953752 4880 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:37.954206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:37.954181 4880 scope.go:122] "RemoveContainer" containerID="d46a25067123d6d611de659ec0444e2ec038629265b823b3dd7a1fc4ea5b5a3f" Oct 31 16:02:37.954390 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:37.954371 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:39.238205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:39.238177 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s7jw9_95a4cede-89fb-421e-be64-69f3ea494179/dns-node-resolver/0.log" Oct 31 16:02:39.308021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:39.307992 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:02:39.308228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:39.308215 4880 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 16:02:39.699696 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:39.699661 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:40.238636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:40.238611 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nmnvx_9cb6cc68-a2af-438f-83be-9f1e9afc57b6/node-ca/0.log" Oct 31 16:02:40.588385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:40.588309 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:41.038283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.038246 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sgkl6_f5e06bd6-7368-4e48-8732-782df89e89dd/serve-healthcheck-canary/0.log" Oct 31 16:02:41.101283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.101247 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:41.262807 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.262774 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:41.639030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.639000 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl_468a8f95-953e-403c-a2d7-931d530da8ee/kube-storage-version-migrator-operator/0.log" Oct 31 16:02:41.959026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.958944 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:41.969940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.969913 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:41.970032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.969947 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:41.970032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.969965 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:41.970160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970064 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:41.970160 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970071 4880 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 16:02:41.970160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970087 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:41.970160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970106 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:41.970160 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970112 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:41.970160 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970124 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-98675d9cb-646m2: secret "image-registry-tls" not found Oct 31 16:02:41.970160 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970070 4880 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970168 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.97012307 +0000 UTC m=+17.005417868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : secret "metrics-daemon-secret" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970195 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls podName:40737053-5dda-4724-be16-bcdeca25cfec nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.970179615 +0000 UTC m=+17.005474407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls") pod "image-registry-98675d9cb-646m2" (UID: "40737053-5dda-4724-be16-bcdeca25cfec") : secret "image-registry-tls" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970217 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970220 4880 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970242 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls podName:0d948a90-6dab-4c2e-b2a3-6815efc5b06e nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.970226782 +0000 UTC m=+17.005521573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls") pod "dns-default-b2gc9" (UID: "0d948a90-6dab-4c2e-b2a3-6815efc5b06e") : secret "dns-default-metrics-tls" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970268 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls podName:4fa7d16a-1eba-4ab3-88f6-d6ae9c638122 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.97025709 +0000 UTC m=+17.005551875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kns59" (UID: "4fa7d16a-1eba-4ab3-88f6-d6ae9c638122") : secret "cluster-monitoring-operator-tls" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970280 4880 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970304 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.970288402 +0000 UTC m=+17.005583203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : configmap references non-existent config key: service-ca.crt Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970335 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls podName:be3d02e8-cb60-4762-b79f-e29afc5b53ac nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.97032289 +0000 UTC m=+17.005617676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-xsxlw" (UID: "be3d02e8-cb60-4762-b79f-e29afc5b53ac") : secret "samples-operator-tls" not found Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970360 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970390 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970424 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:41.970484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970474 4880 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 16:02:41.970910 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970496 4880 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 16:02:41.970910 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970508 4880 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9f8d5c7-c4czr: secret "image-registry-tls" not found Oct 31 16:02:41.970910 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970536 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs podName:efec2af7-bad8-43d3-8fd8-45071136bdcf nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.970523522 +0000 UTC m=+17.005818310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs") pod "router-default-746fc4dd59-qfxr9" (UID: "efec2af7-bad8-43d3-8fd8-45071136bdcf") : secret "router-metrics-certs-default" not found Oct 31 16:02:41.970910 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970542 4880 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 16:02:41.970910 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970553 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls podName:5b7f8b69-c256-492d-8b53-6647fc0ca0e1 nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.970544675 +0000 UTC m=+17.005839468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls") pod "image-registry-7b9f8d5c7-c4czr" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1") : secret "image-registry-tls" not found Oct 31 16:02:41.970910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.970547 4880 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 16:02:41.970910 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:41.970568 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert podName:db4c92c3-fe37-4fbf-be54-a0196309c59c nodeName:}" failed. No retries permitted until 2026-10-31 16:02:49.970560819 +0000 UTC m=+17.005855601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-m76mb" (UID: "db4c92c3-fe37-4fbf-be54-a0196309c59c") : secret "networking-console-plugin-cert" not found Oct 31 16:02:41.972624 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.972603 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68b75f93-390b-4813-9264-a9b9709fc330-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vs98d\" (UID: \"68b75f93-390b-4813-9264-a9b9709fc330\") " pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:41.990073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:41.990036 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vs98d" Oct 31 16:02:42.348050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:42.347971 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vs98d"] Oct 31 16:02:42.351980 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:42.351937 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:42.384873 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:42.384842 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b75f93_390b_4813_9264_a9b9709fc330.slice/crio-fa8cabcaa9f88f0039454c8a64e221c5c3bd37806aacf04b606dccfe548e6a12 WatchSource:0}: Error finding container fa8cabcaa9f88f0039454c8a64e221c5c3bd37806aacf04b606dccfe548e6a12: Status 404 returned error can't find the container with id fa8cabcaa9f88f0039454c8a64e221c5c3bd37806aacf04b606dccfe548e6a12 Oct 31 16:02:42.441586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:42.441566 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:02:42.550475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:42.550443 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:42.744068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:42.744036 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:43.154568 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:43.154541 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:43.168350 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:43.168325 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs98d" event={"ID":"68b75f93-390b-4813-9264-a9b9709fc330","Type":"ContainerStarted","Data":"5f46ba29fbeecc9a301d7941345b04fe02f192cb8f4b6ade64a5b8616eb06dbd"} Oct 31 16:02:43.168476 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:43.168374 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs98d" event={"ID":"68b75f93-390b-4813-9264-a9b9709fc330","Type":"ContainerStarted","Data":"fa8cabcaa9f88f0039454c8a64e221c5c3bd37806aacf04b606dccfe548e6a12"} Oct 31 16:02:43.947927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:43.947850 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:44.108327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:44.108286 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:02:44.172877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:44.172849 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs98d" event={"ID":"68b75f93-390b-4813-9264-a9b9709fc330","Type":"ContainerStarted","Data":"9bf2980598c4add2e9bd72ea8f7e0aed203c4b334f0fa8dbd82d53392b04122d"} Oct 31 16:02:44.341674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:44.341590 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:44.551407 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:44.551374 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:44.753731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:44.753704 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:44.907902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:44.907868 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:45.155042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:45.155004 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:45.297910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:45.297878 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:45.954558 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:45.954524 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:46.100320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:46.100291 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:46.180323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:46.180289 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vs98d" event={"ID":"68b75f93-390b-4813-9264-a9b9709fc330","Type":"ContainerStarted","Data":"9a4c36f1f0c7fa02fde7bde31f5706b0d9fca5ab05df007469af9fe2350ab61b"} Oct 31 16:02:46.346925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:46.346848 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:46.548944 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:46.548910 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:46.749696 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:46.749668 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:47.083561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.083482 4880 scope.go:122] "RemoveContainer" containerID="d46a25067123d6d611de659ec0444e2ec038629265b823b3dd7a1fc4ea5b5a3f" Oct 31 16:02:47.151198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.151166 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:47.185876 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.185847 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/2.log" Oct 31 16:02:47.186298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.185942 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" event={"ID":"e9cad0b0-2b17-48f6-9c14-12a419791b1f","Type":"ContainerStarted","Data":"95ecf8e08d65575dbc75cb66b1eefaa46ae13a07071d5c02c668c587f1e7fc4b"} Oct 31 16:02:47.186471 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.186449 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:47.187935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.187911 4880 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-bthm6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Oct 31 16:02:47.188016 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.187991 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Oct 31 16:02:47.697981 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.697948 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:47.947524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.947490 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:47.954977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.954925 4880 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-bthm6 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Oct 31 16:02:47.954977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:47.954968 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Oct 31 16:02:48.191011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.190983 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:02:48.191420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.191349 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/2.log" Oct 31 16:02:48.191420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.191378 4880 generic.go:356] "Generic (PLEG): container finished" podID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerID="95ecf8e08d65575dbc75cb66b1eefaa46ae13a07071d5c02c668c587f1e7fc4b" exitCode=255 Oct 31 16:02:48.191420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.191411 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" event={"ID":"e9cad0b0-2b17-48f6-9c14-12a419791b1f","Type":"ContainerDied","Data":"95ecf8e08d65575dbc75cb66b1eefaa46ae13a07071d5c02c668c587f1e7fc4b"} Oct 31 16:02:48.191517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.191451 4880 scope.go:122] "RemoveContainer" containerID="d46a25067123d6d611de659ec0444e2ec038629265b823b3dd7a1fc4ea5b5a3f" Oct 31 16:02:48.191717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.191698 4880 scope.go:122] "RemoveContainer" containerID="95ecf8e08d65575dbc75cb66b1eefaa46ae13a07071d5c02c668c587f1e7fc4b" Oct 31 16:02:48.191903 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:48.191886 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:48.355041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.354958 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:48.553088 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.553040 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:48.748283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:48.748249 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:49.146340 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:49.146310 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:49.195708 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:49.195686 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:02:49.196081 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:49.196067 4880 scope.go:122] "RemoveContainer" containerID="95ecf8e08d65575dbc75cb66b1eefaa46ae13a07071d5c02c668c587f1e7fc4b" Oct 31 16:02:49.196324 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:49.196293 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:49.860019 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:49.859985 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-98675d9cb-646m2"] Oct 31 16:02:49.860239 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:49.860221 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-98675d9cb-646m2" podUID="40737053-5dda-4724-be16-bcdeca25cfec" Oct 31 16:02:49.956629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:49.956595 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:50.048441 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048411 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:02:50.048441 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048446 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:50.048683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048464 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:50.048683 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:50.048556 4880 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 16:02:50.048683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048590 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:50.048683 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:50.048613 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs podName:55a6396f-e3a0-42ec-88fa-c4d23384e807 nodeName:}" failed. No retries permitted until 2026-10-31 16:03:06.04859956 +0000 UTC m=+33.083894347 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs") pod "network-metrics-daemon-mmp6g" (UID: "55a6396f-e3a0-42ec-88fa-c4d23384e807") : secret "metrics-daemon-secret" not found Oct 31 16:02:50.048683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048639 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:50.048683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048683 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:50.048989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048721 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:50.048989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.048762 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:50.049093 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.049051 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:50.049542 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.049516 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efec2af7-bad8-43d3-8fd8-45071136bdcf-service-ca-bundle\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:50.051809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.051788 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efec2af7-bad8-43d3-8fd8-45071136bdcf-metrics-certs\") pod \"router-default-746fc4dd59-qfxr9\" (UID: \"efec2af7-bad8-43d3-8fd8-45071136bdcf\") " pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:50.051933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.051828 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fa7d16a-1eba-4ab3-88f6-d6ae9c638122-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kns59\" (UID: \"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:50.051933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.051887 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/be3d02e8-cb60-4762-b79f-e29afc5b53ac-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-xsxlw\" (UID: \"be3d02e8-cb60-4762-b79f-e29afc5b53ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:50.051933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.051929 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"image-registry-7b9f8d5c7-c4czr\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:50.052049 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.051971 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/db4c92c3-fe37-4fbf-be54-a0196309c59c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-m76mb\" (UID: \"db4c92c3-fe37-4fbf-be54-a0196309c59c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:50.052231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.052207 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"image-registry-98675d9cb-646m2\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:50.052355 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.052337 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d948a90-6dab-4c2e-b2a3-6815efc5b06e-metrics-tls\") pod \"dns-default-b2gc9\" (UID: \"0d948a90-6dab-4c2e-b2a3-6815efc5b06e\") " pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:50.079197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.079170 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" Oct 31 16:02:50.079310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.079275 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" Oct 31 16:02:50.081297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.081277 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:50.082088 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.082072 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" Oct 31 16:02:50.088535 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.088518 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:50.089598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.089578 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:50.106872 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.106759 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:50.203370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.203343 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:50.213948 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.213818 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:50.346453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.346424 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:50.354283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354260 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354301 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354323 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354341 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354460 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354508 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354550 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354668 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\") pod \"40737053-5dda-4724-be16-bcdeca25cfec\" (UID: \"40737053-5dda-4724-be16-bcdeca25cfec\") " Oct 31 16:02:50.354792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354767 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:02:50.354883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354775 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:02:50.355063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354964 4880 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-trusted-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.355063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.354988 4880 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/40737053-5dda-4724-be16-bcdeca25cfec-ca-trust-extracted\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.355211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.355071 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:02:50.356648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.356628 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:02:50.356743 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.356664 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:02:50.356805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.356768 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "kube-api-access-6hm7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:02:50.356997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.356978 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:02:50.357052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.357018 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets" pod "40737053-5dda-4724-be16-bcdeca25cfec" (UID: "40737053-5dda-4724-be16-bcdeca25cfec"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:02:50.455956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.455892 4880 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-bound-sa-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.455956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.455919 4880 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-installation-pull-secrets\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.455956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.455929 4880 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/40737053-5dda-4724-be16-bcdeca25cfec-registry-certificates\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.455956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.455940 4880 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/40737053-5dda-4724-be16-bcdeca25cfec-image-registry-private-configuration\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.455956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.455950 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hm7f\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-kube-api-access-6hm7f\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.455956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.455960 4880 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/40737053-5dda-4724-be16-bcdeca25cfec-registry-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:02:50.485342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.485311 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kns59"] Oct 31 16:02:50.498499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.498470 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:50.526540 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:50.526509 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa7d16a_1eba_4ab3_88f6_d6ae9c638122.slice/crio-0a1e3f988f51f404e89ac4a522e54d7915881959d615b074eba6fbef99ab66c7 WatchSource:0}: Error finding container 0a1e3f988f51f404e89ac4a522e54d7915881959d615b074eba6fbef99ab66c7: Status 404 returned error can't find the container with id 0a1e3f988f51f404e89ac4a522e54d7915881959d615b074eba6fbef99ab66c7 Oct 31 16:02:50.534497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.534472 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-746fc4dd59-qfxr9"] Oct 31 16:02:50.537769 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:50.537743 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefec2af7_bad8_43d3_8fd8_45071136bdcf.slice/crio-715efd8d7e790f91a19870a49a631ba1886adbb26036055e4ed4da8515e7c6ef WatchSource:0}: Error finding container 715efd8d7e790f91a19870a49a631ba1886adbb26036055e4ed4da8515e7c6ef: Status 404 returned error can't find the container with id 715efd8d7e790f91a19870a49a631ba1886adbb26036055e4ed4da8515e7c6ef Oct 31 16:02:50.579825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.579805 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2gc9"] Oct 31 16:02:50.582830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.582748 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw"] Oct 31 16:02:50.583486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.583421 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-m76mb"] Oct 31 16:02:50.584047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.584014 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b9f8d5c7-c4czr"] Oct 31 16:02:50.584161 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:50.584096 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d948a90_6dab_4c2e_b2a3_6815efc5b06e.slice/crio-64a7ba764380ac4a9c02778634e967df9aa61f9bd92ddab2f21aef8ec359f2e3 WatchSource:0}: Error finding container 64a7ba764380ac4a9c02778634e967df9aa61f9bd92ddab2f21aef8ec359f2e3: Status 404 returned error can't find the container with id 64a7ba764380ac4a9c02778634e967df9aa61f9bd92ddab2f21aef8ec359f2e3 Oct 31 16:02:50.588774 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:50.588746 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4c92c3_fe37_4fbf_be54_a0196309c59c.slice/crio-ded1836a9bca944e7578413e9894fd22c45ad29051faf82e72be3b6519df1c8d WatchSource:0}: Error finding container ded1836a9bca944e7578413e9894fd22c45ad29051faf82e72be3b6519df1c8d: Status 404 returned error can't find the container with id ded1836a9bca944e7578413e9894fd22c45ad29051faf82e72be3b6519df1c8d Oct 31 16:02:50.589469 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:02:50.589355 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7f8b69_c256_492d_8b53_6647fc0ca0e1.slice/crio-4d3d6a6578a92b19773cdd8014a7f66802f78e698aa1fd21e89f2b549778a58b WatchSource:0}: Error finding container 4d3d6a6578a92b19773cdd8014a7f66802f78e698aa1fd21e89f2b549778a58b: Status 404 returned error can't find the container with id 4d3d6a6578a92b19773cdd8014a7f66802f78e698aa1fd21e89f2b549778a58b Oct 31 16:02:50.645296 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.645270 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:50.801645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:50.801559 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:51.148950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.148920 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:51.207883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.207847 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2gc9" event={"ID":"0d948a90-6dab-4c2e-b2a3-6815efc5b06e","Type":"ContainerStarted","Data":"64a7ba764380ac4a9c02778634e967df9aa61f9bd92ddab2f21aef8ec359f2e3"} Oct 31 16:02:51.208941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.208919 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" event={"ID":"db4c92c3-fe37-4fbf-be54-a0196309c59c","Type":"ContainerStarted","Data":"ded1836a9bca944e7578413e9894fd22c45ad29051faf82e72be3b6519df1c8d"} Oct 31 16:02:51.210338 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.210317 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" event={"ID":"efec2af7-bad8-43d3-8fd8-45071136bdcf","Type":"ContainerStarted","Data":"6268988f39ea50d3343b2a47bafb186e02c2505d161f7489d9cd98ab0745d0bd"} Oct 31 16:02:51.210418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.210353 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" event={"ID":"efec2af7-bad8-43d3-8fd8-45071136bdcf","Type":"ContainerStarted","Data":"715efd8d7e790f91a19870a49a631ba1886adbb26036055e4ed4da8515e7c6ef"} Oct 31 16:02:51.211747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.211727 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" event={"ID":"5b7f8b69-c256-492d-8b53-6647fc0ca0e1","Type":"ContainerStarted","Data":"6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f"} Oct 31 16:02:51.211832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.211762 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" event={"ID":"5b7f8b69-c256-492d-8b53-6647fc0ca0e1","Type":"ContainerStarted","Data":"4d3d6a6578a92b19773cdd8014a7f66802f78e698aa1fd21e89f2b549778a58b"} Oct 31 16:02:51.211832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.211786 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:02:51.212785 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.212762 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" event={"ID":"be3d02e8-cb60-4762-b79f-e29afc5b53ac","Type":"ContainerStarted","Data":"32736594e4d557bdbcd46ff71f86208baeb22d876c909b2dbd2c7b4075e5cbd0"} Oct 31 16:02:51.213860 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.213842 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" event={"ID":"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122","Type":"ContainerStarted","Data":"0a1e3f988f51f404e89ac4a522e54d7915881959d615b074eba6fbef99ab66c7"} Oct 31 16:02:51.213942 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.213865 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-98675d9cb-646m2" Oct 31 16:02:51.349509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.349429 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:51.478859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.478828 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-98675d9cb-646m2"] Oct 31 16:02:51.482231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:51.482177 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-98675d9cb-646m2"] Oct 31 16:02:52.082099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:52.081886 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:52.087381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:52.087358 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:52.218828 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:52.218799 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:52.220370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:52.220345 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-746fc4dd59-qfxr9" Oct 31 16:02:53.089264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:53.089224 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40737053-5dda-4724-be16-bcdeca25cfec" path="/var/lib/kubelet/pods/40737053-5dda-4724-be16-bcdeca25cfec/volumes" Oct 31 16:02:53.227548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:53.227516 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-m76mb" event={"ID":"db4c92c3-fe37-4fbf-be54-a0196309c59c","Type":"ContainerStarted","Data":"1679f6bb932e657cd002e80ba5a2c2cc75896d2a64dd3d6f3dad479f9ddb6431"} Oct 31 16:02:54.029668 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.029636 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4wccg" Oct 31 16:02:54.232732 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.232697 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" event={"ID":"be3d02e8-cb60-4762-b79f-e29afc5b53ac","Type":"ContainerStarted","Data":"fc8ed37c601383931d813eb3da4afee505e10818feb446795e893db6aff724f0"} Oct 31 16:02:54.233178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.232746 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-xsxlw" event={"ID":"be3d02e8-cb60-4762-b79f-e29afc5b53ac","Type":"ContainerStarted","Data":"05551bc44ab36326c0747a841eee8b95aece94dbbb10d8f265fef1bcd194fa7d"} Oct 31 16:02:54.234174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.234147 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kns59" event={"ID":"4fa7d16a-1eba-4ab3-88f6-d6ae9c638122","Type":"ContainerStarted","Data":"87c506163f3cac39b1fea1dac5865c539c5472c2069e27bad6d834e138dabcfc"} Oct 31 16:02:54.235789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.235758 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2gc9" event={"ID":"0d948a90-6dab-4c2e-b2a3-6815efc5b06e","Type":"ContainerStarted","Data":"6f50cf373dad51ad4e5238ac7e676858038c3088945ae82953c509fa5bd1becc"} Oct 31 16:02:54.235884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.235812 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2gc9" event={"ID":"0d948a90-6dab-4c2e-b2a3-6815efc5b06e","Type":"ContainerStarted","Data":"c432c65d7e3853b2a1dd3bbdcd8e64e1cc5ebd241b81a00b8dedc706ef02e74a"} Oct 31 16:02:54.408010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.407973 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:54.552865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.552827 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:54.761228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:54.761118 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:55.239406 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:55.239379 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b2gc9" Oct 31 16:02:55.386278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:55.386246 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:55.776466 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:55.776428 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:57.953845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:57.953795 4880 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:02:57.954253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:57.954239 4880 scope.go:122] "RemoveContainer" containerID="95ecf8e08d65575dbc75cb66b1eefaa46ae13a07071d5c02c668c587f1e7fc4b" Oct 31 16:02:57.954424 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:02:57.954405 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-bthm6_openshift-console-operator(e9cad0b0-2b17-48f6-9c14-12a419791b1f)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" Oct 31 16:02:58.640420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:58.640387 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:58.841904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:58.841870 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:58.992766 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:58.992658 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:02:59.142927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:02:59.142890 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:00.518985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:00.518950 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:00.710046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:00.710007 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:01.051203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:01.051176 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m6xpv" Oct 31 16:03:03.879794 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:03:03.879756 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:03:04.110790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:04.110764 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kddfn" Oct 31 16:03:04.390117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:04.390077 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:04.775036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:04.774961 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:04.976303 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:04.976268 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:05.230954 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.230921 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:05.244296 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.244277 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b2gc9" Oct 31 16:03:05.391445 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.391413 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:05.512591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.512522 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qv4ch"] Oct 31 16:03:05.539427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.539382 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:05.547574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.547554 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578449 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-accelerators-collector-config\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578596 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578479 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7kw\" (UniqueName: \"kubernetes.io/projected/49f7f8e5-de0c-4c95-8c4e-2accedb15997-kube-api-access-9r7kw\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578596 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578500 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-tls\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578596 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578523 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49f7f8e5-de0c-4c95-8c4e-2accedb15997-metrics-client-ca\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578688 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-root\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578731 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-textfile\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578837 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578916 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578870 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-wtmp\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.578971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.578911 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-sys\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.595859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.595835 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 16:03:05.595979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.595958 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 16:03:05.596045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.596027 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 16:03:05.596117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.596103 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 16:03:05.596198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.596171 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jlhfg\"" Oct 31 16:03:05.679562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679535 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-sys\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.679706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679570 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-accelerators-collector-config\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.679706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679591 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7kw\" (UniqueName: \"kubernetes.io/projected/49f7f8e5-de0c-4c95-8c4e-2accedb15997-kube-api-access-9r7kw\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.679706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679611 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-tls\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.679706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679647 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49f7f8e5-de0c-4c95-8c4e-2accedb15997-metrics-client-ca\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.679706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679672 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-sys\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.679706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679695 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-root\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679722 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-textfile\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679748 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-root\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679906 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.679942 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-wtmp\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.680053 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-textfile\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.680117 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-wtmp\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.680315 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-accelerators-collector-config\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.680381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.680333 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49f7f8e5-de0c-4c95-8c4e-2accedb15997-metrics-client-ca\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.682027 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.682003 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-tls\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.682197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.682181 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49f7f8e5-de0c-4c95-8c4e-2accedb15997-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.734011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.733983 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7kw\" (UniqueName: \"kubernetes.io/projected/49f7f8e5-de0c-4c95-8c4e-2accedb15997-kube-api-access-9r7kw\") pod \"node-exporter-qv4ch\" (UID: \"49f7f8e5-de0c-4c95-8c4e-2accedb15997\") " pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.755533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.755507 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:05.896750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.896724 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qv4ch" Oct 31 16:03:05.912754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:05.912715 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:05.925335 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:03:05.925312 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f7f8e5_de0c_4c95_8c4e_2accedb15997.slice/crio-51417d28fe7c63b3ecd0bf82baff937b4bb629c327e31e4a454402627ab3637f WatchSource:0}: Error finding container 51417d28fe7c63b3ecd0bf82baff937b4bb629c327e31e4a454402627ab3637f: Status 404 returned error can't find the container with id 51417d28fe7c63b3ecd0bf82baff937b4bb629c327e31e4a454402627ab3637f Oct 31 16:03:06.083402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:06.083369 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:03:06.085648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:06.085630 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55a6396f-e3a0-42ec-88fa-c4d23384e807-metrics-certs\") pod \"network-metrics-daemon-mmp6g\" (UID: \"55a6396f-e3a0-42ec-88fa-c4d23384e807\") " pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:03:06.234477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:06.234395 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:06.276345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:06.276312 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qv4ch" event={"ID":"49f7f8e5-de0c-4c95-8c4e-2accedb15997","Type":"ContainerStarted","Data":"51417d28fe7c63b3ecd0bf82baff937b4bb629c327e31e4a454402627ab3637f"} Oct 31 16:03:06.284493 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:06.284463 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mmp6g" Oct 31 16:03:06.670780 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:06.670748 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mmp6g"] Oct 31 16:03:06.674987 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:03:06.674962 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a6396f_e3a0_42ec_88fa_c4d23384e807.slice/crio-29342242faf3cb58b6f0cfe87376a98a44276c2402ba97f4addfcca78f6d1f1b WatchSource:0}: Error finding container 29342242faf3cb58b6f0cfe87376a98a44276c2402ba97f4addfcca78f6d1f1b: Status 404 returned error can't find the container with id 29342242faf3cb58b6f0cfe87376a98a44276c2402ba97f4addfcca78f6d1f1b Oct 31 16:03:06.818107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:06.818066 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:07.215186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:07.215145 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:07.281732 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:07.281695 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mmp6g" event={"ID":"55a6396f-e3a0-42ec-88fa-c4d23384e807","Type":"ContainerStarted","Data":"29342242faf3cb58b6f0cfe87376a98a44276c2402ba97f4addfcca78f6d1f1b"} Oct 31 16:03:07.671650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:07.671621 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:08.287593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:08.287524 4880 generic.go:356] "Generic (PLEG): container finished" podID="49f7f8e5-de0c-4c95-8c4e-2accedb15997" containerID="a786d4cce103854bc56b8c56c33ed5d78ed22480d7c6bf3ef19ca74bc350c4b2" exitCode=0 Oct 31 16:03:08.287593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:08.287589 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qv4ch" event={"ID":"49f7f8e5-de0c-4c95-8c4e-2accedb15997","Type":"ContainerDied","Data":"a786d4cce103854bc56b8c56c33ed5d78ed22480d7c6bf3ef19ca74bc350c4b2"} Oct 31 16:03:09.293764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.293722 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qv4ch" event={"ID":"49f7f8e5-de0c-4c95-8c4e-2accedb15997","Type":"ContainerStarted","Data":"6e6d1caf60ff874dff22a95ce037536c78d3e123c273f87a45a6d6019ce1e4ac"} Oct 31 16:03:09.294231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.293799 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qv4ch" event={"ID":"49f7f8e5-de0c-4c95-8c4e-2accedb15997","Type":"ContainerStarted","Data":"2abda8d6bf66a86e0b0ada992f0880ad6dccb881990ef77c7eb87d80ca946fb7"} Oct 31 16:03:09.295980 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.295861 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mmp6g" event={"ID":"55a6396f-e3a0-42ec-88fa-c4d23384e807","Type":"ContainerStarted","Data":"d5747ff01033c061501f68b506923285ced4a9b95fedd471d10b614b425eaa61"} Oct 31 16:03:09.295980 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.295913 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mmp6g" event={"ID":"55a6396f-e3a0-42ec-88fa-c4d23384e807","Type":"ContainerStarted","Data":"efac660776d851f442a20529de9544c48409e09b9729697cc398c7451c9b2449"} Oct 31 16:03:09.368690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.368663 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:09.403855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.403812 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qv4ch" podStartSLOduration=2.277904363 podStartE2EDuration="4.40379819s" totalImagesPullingTime="2.032091481s" totalInitContainerRuntime="93.802346ms" isStatefulPod=true podCreationTimestamp="2026-10-31 16:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:09.403357051 +0000 UTC m=+36.438651857" watchObservedRunningTime="2026-10-31 16:03:09.40379819 +0000 UTC m=+36.439093030" Oct 31 16:03:09.530735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.530709 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:09.689514 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.689489 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:09.892046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:09.892015 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:10.082949 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.082869 4880 scope.go:122] "RemoveContainer" containerID="95ecf8e08d65575dbc75cb66b1eefaa46ae13a07071d5c02c668c587f1e7fc4b" Oct 31 16:03:10.122578 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.122552 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:10.305265 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.305237 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:03:10.305677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.305351 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" event={"ID":"e9cad0b0-2b17-48f6-9c14-12a419791b1f","Type":"ContainerStarted","Data":"a52ef8eb37ed8c14f6717e7e0d92228aa33118bd7aee40b3332c58b48d14aba7"} Oct 31 16:03:10.305677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.305625 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:03:10.306737 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.306714 4880 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-bthm6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Oct 31 16:03:10.306845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.306752 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Oct 31 16:03:10.374234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:10.374150 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:11.315382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:11.315352 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" Oct 31 16:03:11.541783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:11.541750 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:11.813382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:11.813350 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:11.964863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:11.964805 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:12.116205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.116109 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:12.223450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.223421 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:03:12.267450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.267413 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b9f8d5c7-c4czr"] Oct 31 16:03:12.385741 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.385709 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:12.528244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.528205 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:12.681464 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.681380 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:12.815904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.815823 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-28g9d"] Oct 31 16:03:12.818081 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.818060 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:12.866770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.866741 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 16:03:12.866966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.866827 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 16:03:12.866966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.866897 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-lfxjg\"" Oct 31 16:03:12.886073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.886042 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:12.913719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.913688 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-28g9d"] Oct 31 16:03:12.946310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.946223 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4jr\" (UniqueName: \"kubernetes.io/projected/44223443-5020-4634-8e24-4fa00a2cb80e-kube-api-access-5t4jr\") pod \"downloads-6dd789c8b7-28g9d\" (UID: \"44223443-5020-4634-8e24-4fa00a2cb80e\") " pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:12.946464 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:12.946345 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44223443-5020-4634-8e24-4fa00a2cb80e-tmp\") pod \"downloads-6dd789c8b7-28g9d\" (UID: \"44223443-5020-4634-8e24-4fa00a2cb80e\") " pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:13.034589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:13.034552 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:13.047616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:13.047584 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44223443-5020-4634-8e24-4fa00a2cb80e-tmp\") pod \"downloads-6dd789c8b7-28g9d\" (UID: \"44223443-5020-4634-8e24-4fa00a2cb80e\") " pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:13.047752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:13.047626 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4jr\" (UniqueName: \"kubernetes.io/projected/44223443-5020-4634-8e24-4fa00a2cb80e-kube-api-access-5t4jr\") pod \"downloads-6dd789c8b7-28g9d\" (UID: \"44223443-5020-4634-8e24-4fa00a2cb80e\") " pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:13.047973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:13.047955 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44223443-5020-4634-8e24-4fa00a2cb80e-tmp\") pod \"downloads-6dd789c8b7-28g9d\" (UID: \"44223443-5020-4634-8e24-4fa00a2cb80e\") " pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:13.100096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:13.100060 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4jr\" (UniqueName: \"kubernetes.io/projected/44223443-5020-4634-8e24-4fa00a2cb80e-kube-api-access-5t4jr\") pod \"downloads-6dd789c8b7-28g9d\" (UID: \"44223443-5020-4634-8e24-4fa00a2cb80e\") " pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:13.127915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:13.127879 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:13.582295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:13.582262 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-28g9d"] Oct 31 16:03:13.585435 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:03:13.585408 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44223443_5020_4634_8e24_4fa00a2cb80e.slice/crio-9abbe8d526ed139a874a724f9e20a397ca343afb9ca5ce9094cfbd925e23c933 WatchSource:0}: Error finding container 9abbe8d526ed139a874a724f9e20a397ca343afb9ca5ce9094cfbd925e23c933: Status 404 returned error can't find the container with id 9abbe8d526ed139a874a724f9e20a397ca343afb9ca5ce9094cfbd925e23c933 Oct 31 16:03:14.288527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:14.288486 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:14.323370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:14.323338 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-28g9d" event={"ID":"44223443-5020-4634-8e24-4fa00a2cb80e","Type":"ContainerStarted","Data":"9abbe8d526ed139a874a724f9e20a397ca343afb9ca5ce9094cfbd925e23c933"} Oct 31 16:03:15.402294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:15.402251 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:15.547097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:15.547061 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:17.553515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:17.553479 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:17.701427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:17.701394 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:17.863992 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:17.863866 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:20.050886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:20.050799 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:20.193930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:20.193895 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:20.451768 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:20.451669 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:20.607699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:20.607659 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:20.775194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:20.775109 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:22.174303 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:22.174261 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:22.327113 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:22.327076 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:23.426571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:23.426528 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:23.652623 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:23.652593 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:30.022020 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:30.021978 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:30.169817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:30.169782 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:30.401501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:30.401459 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-28g9d" event={"ID":"44223443-5020-4634-8e24-4fa00a2cb80e","Type":"ContainerStarted","Data":"0e17a955826058a3ac85b57f8967c65d83dc87faee3cca28e8d01cf8ab9c9ee4"} Oct 31 16:03:30.401692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:30.401641 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:30.417030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:30.416988 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-28g9d" Oct 31 16:03:30.600978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:30.600916 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-28g9d" podStartSLOduration=2.7744799049999997 podStartE2EDuration="18.60089912s" totalImagesPullingTime="15.826419215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:03:30.599785139 +0000 UTC m=+57.635079944" watchObservedRunningTime="2026-10-31 16:03:30.60089912 +0000 UTC m=+57.636193929" Oct 31 16:03:33.884808 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:03:33.884765 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:03:35.457451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:35.457409 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:35.611804 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:35.611763 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:35.990024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:35.989984 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:36.144500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:36.144467 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:37.333978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.333936 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" podUID="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" containerName="registry" containerID="cri-o://6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f" gracePeriod=30 Oct 31 16:03:37.628338 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.628300 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:03:37.677483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677449 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677499 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677542 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677572 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677596 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677627 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677683 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677715 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\") pod \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\" (UID: \"5b7f8b69-c256-492d-8b53-6647fc0ca0e1\") " Oct 31 16:03:37.677920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677860 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:03:37.678060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.677953 4880 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-trusted-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.679119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.679084 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:03:37.680704 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.680670 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:03:37.681123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.681086 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:03:37.682190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.682165 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "kube-api-access-vb5g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:03:37.682625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.682589 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:03:37.682814 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.682790 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:03:37.686789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.686766 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted" pod "5b7f8b69-c256-492d-8b53-6647fc0ca0e1" (UID: "5b7f8b69-c256-492d-8b53-6647fc0ca0e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:03:37.778416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.778377 4880 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-certificates\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.778416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.778412 4880 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-ca-trust-extracted\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.778416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.778423 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5g9\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-kube-api-access-vb5g9\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.778659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.778431 4880 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-installation-pull-secrets\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.778659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.778441 4880 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-image-registry-private-configuration\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.778659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.778458 4880 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-registry-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:37.778659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:37.778470 4880 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5b7f8b69-c256-492d-8b53-6647fc0ca0e1-bound-sa-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:03:38.435742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.435707 4880 generic.go:356] "Generic (PLEG): container finished" podID="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" containerID="6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f" exitCode=0 Oct 31 16:03:38.436185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.435759 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" Oct 31 16:03:38.436185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.435794 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" event={"ID":"5b7f8b69-c256-492d-8b53-6647fc0ca0e1","Type":"ContainerDied","Data":"6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f"} Oct 31 16:03:38.436185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.435850 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9f8d5c7-c4czr" event={"ID":"5b7f8b69-c256-492d-8b53-6647fc0ca0e1","Type":"ContainerDied","Data":"4d3d6a6578a92b19773cdd8014a7f66802f78e698aa1fd21e89f2b549778a58b"} Oct 31 16:03:38.436185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.435876 4880 scope.go:122] "RemoveContainer" containerID="6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f" Oct 31 16:03:38.444329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.444312 4880 scope.go:122] "RemoveContainer" containerID="6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f" Oct 31 16:03:38.444577 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:03:38.444558 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f\": container with ID starting with 6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f not found: ID does not exist" containerID="6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f" Oct 31 16:03:38.444641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.444590 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f"} err="failed to get container status \"6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f\": rpc error: code = NotFound desc = could not find container \"6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f\": container with ID starting with 6e757915d3deefd9d9619706d733a232031d20ba07d6995f1a3702bf0a752c2f not found: ID does not exist" Oct 31 16:03:38.637056 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.637020 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b9f8d5c7-c4czr"] Oct 31 16:03:38.683837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:38.683738 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b9f8d5c7-c4czr"] Oct 31 16:03:39.087578 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:39.087548 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" path="/var/lib/kubelet/pods/5b7f8b69-c256-492d-8b53-6647fc0ca0e1/volumes" Oct 31 16:03:44.205485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:44.205450 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:44.344800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:44.344767 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:50.456542 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:50.456505 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:50.600540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:50.600503 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:52.024152 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:52.024114 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:52.169214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:52.169178 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:53.222569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:53.222531 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:53.362247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:53.362216 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:53.506024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:53.505954 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:55.839243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:55.839212 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:03:55.987862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:03:55.987824 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:02.948753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:02.948716 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:03.141259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:03.141226 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:03.879685 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:04:03.879648 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:04:04.525879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:04.525851 4880 generic.go:356] "Generic (PLEG): container finished" podID="468a8f95-953e-403c-a2d7-931d530da8ee" containerID="3151abc0569db732c20a7841e9bec5237e213d3910f245e58a2e077cbe9927fd" exitCode=0 Oct 31 16:04:04.526260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:04.525910 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" event={"ID":"468a8f95-953e-403c-a2d7-931d530da8ee","Type":"ContainerDied","Data":"3151abc0569db732c20a7841e9bec5237e213d3910f245e58a2e077cbe9927fd"} Oct 31 16:04:04.526260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:04.526246 4880 scope.go:122] "RemoveContainer" containerID="3151abc0569db732c20a7841e9bec5237e213d3910f245e58a2e077cbe9927fd" Oct 31 16:04:05.458578 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:05.458547 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:05.530632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:05.530600 4880 generic.go:356] "Generic (PLEG): container finished" podID="44e3677f-537d-4cd9-8b72-3486e64246b6" containerID="d2b2b1099e9bf45b1faa743c1b64df77eac320008c776e8debca4a5a2ee6fb8a" exitCode=0 Oct 31 16:04:05.531038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:05.530678 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" event={"ID":"44e3677f-537d-4cd9-8b72-3486e64246b6","Type":"ContainerDied","Data":"d2b2b1099e9bf45b1faa743c1b64df77eac320008c776e8debca4a5a2ee6fb8a"} Oct 31 16:04:05.531100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:05.531040 4880 scope.go:122] "RemoveContainer" containerID="d2b2b1099e9bf45b1faa743c1b64df77eac320008c776e8debca4a5a2ee6fb8a" Oct 31 16:04:05.532654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:05.532627 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-dfmzl" event={"ID":"468a8f95-953e-403c-a2d7-931d530da8ee","Type":"ContainerStarted","Data":"1d3640327304e2b269ed22832112b2fe7c5dfcdadf47a6f97a1680b1d79e824c"} Oct 31 16:04:05.612716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:05.612684 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:06.537449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:06.537407 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-6srcs" event={"ID":"44e3677f-537d-4cd9-8b72-3486e64246b6","Type":"ContainerStarted","Data":"1c567559589f658be0141249522f6b4a60894bfd1cb085820e2c94ad5f4fc227"} Oct 31 16:04:10.551035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:10.551002 4880 generic.go:356] "Generic (PLEG): container finished" podID="fb4e8bad-6ee9-4270-a117-70210189abe1" containerID="5260060bac58f587fa611531df51b58e35a8996271a2ab2cbf7c8c4b4d5f5d88" exitCode=0 Oct 31 16:04:10.551477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:10.551074 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9zgx8" event={"ID":"fb4e8bad-6ee9-4270-a117-70210189abe1","Type":"ContainerDied","Data":"5260060bac58f587fa611531df51b58e35a8996271a2ab2cbf7c8c4b4d5f5d88"} Oct 31 16:04:10.551477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:10.551417 4880 scope.go:122] "RemoveContainer" containerID="5260060bac58f587fa611531df51b58e35a8996271a2ab2cbf7c8c4b4d5f5d88" Oct 31 16:04:11.556547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.556510 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9zgx8" event={"ID":"fb4e8bad-6ee9-4270-a117-70210189abe1","Type":"ContainerStarted","Data":"d9e512d1fbab5c950015f211e93bc8fe04d1567bfcb922e593e480ed96802df9"} Oct 31 16:04:11.810875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.810803 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hlqrl-22v6k"] Oct 31 16:04:11.811107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.811095 4880 cpu_manager.go:414] "containerMap: removing container" podUID="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" containerName="registry" Oct 31 16:04:11.811168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.811108 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" containerName="registry" Oct 31 16:04:11.811206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.811192 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7f8b69-c256-492d-8b53-6647fc0ca0e1" containerName="registry" Oct 31 16:04:11.812773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.812757 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:11.865450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.865415 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hlqrl-22v6k"] Oct 31 16:04:11.940414 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.940384 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-service-ca-bundle\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:11.940553 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.940418 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hts5n\" (UniqueName: \"kubernetes.io/projected/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-kube-api-access-hts5n\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:11.940553 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.940443 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-archives-path\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:11.940553 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:11.940498 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-trusted-ca-bundle\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.040933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.040902 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-archives-path\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.040933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.040932 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-trusted-ca-bundle\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.041106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.040982 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-service-ca-bundle\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.041106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.041001 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hts5n\" (UniqueName: \"kubernetes.io/projected/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-kube-api-access-hts5n\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.041316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.041296 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-archives-path\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.041653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.041636 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-service-ca-bundle\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.041721 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.041704 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-trusted-ca-bundle\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.094175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.094098 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hts5n\" (UniqueName: \"kubernetes.io/projected/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-kube-api-access-hts5n\") pod \"periodic-gathering-hlqrl-22v6k\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.123039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.123015 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:04:12.480303 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.480275 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hlqrl-22v6k"] Oct 31 16:04:12.483509 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:04:12.483478 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa967dc1_1a67_4042_ab72_7eecf73eb5ee.slice/crio-4ee538e8782e916d99a87faa20afc2d06e0c565cdc1bd133de1b68c018b36976 WatchSource:0}: Error finding container 4ee538e8782e916d99a87faa20afc2d06e0c565cdc1bd133de1b68c018b36976: Status 404 returned error can't find the container with id 4ee538e8782e916d99a87faa20afc2d06e0c565cdc1bd133de1b68c018b36976 Oct 31 16:04:12.561833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.561799 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" event={"ID":"fa967dc1-1a67-4042-ab72-7eecf73eb5ee","Type":"ContainerStarted","Data":"a3f10dfbe3b3b70cd0f01eb0ba3eb43d2140d00dec3476cdcdd5b01b973d8bcc"} Oct 31 16:04:12.562210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.561853 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" event={"ID":"fa967dc1-1a67-4042-ab72-7eecf73eb5ee","Type":"ContainerStarted","Data":"4ee538e8782e916d99a87faa20afc2d06e0c565cdc1bd133de1b68c018b36976"} Oct 31 16:04:12.666963 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:12.666903 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" podStartSLOduration=1.6668897280000001 podStartE2EDuration="1.666889728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:04:12.666030629 +0000 UTC m=+99.701325456" watchObservedRunningTime="2026-10-31 16:04:12.666889728 +0000 UTC m=+99.702184534" Oct 31 16:04:23.760450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:04:23.760412 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:04:33.879709 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:04:33.879671 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:05:03.881394 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:05:03.881352 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:05:24.082402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:24.082361 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:33.880360 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:05:33.880317 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:05:37.859618 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:37.859586 4880 generic.go:356] "Generic (PLEG): container finished" podID="fa967dc1-1a67-4042-ab72-7eecf73eb5ee" containerID="a3f10dfbe3b3b70cd0f01eb0ba3eb43d2140d00dec3476cdcdd5b01b973d8bcc" exitCode=0 Oct 31 16:05:37.860065 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:37.859629 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" event={"ID":"fa967dc1-1a67-4042-ab72-7eecf73eb5ee","Type":"ContainerDied","Data":"a3f10dfbe3b3b70cd0f01eb0ba3eb43d2140d00dec3476cdcdd5b01b973d8bcc"} Oct 31 16:05:38.992810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:38.992782 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:05:39.135594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.135561 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-kube-api-access-hts5n\" (UniqueName: \"kubernetes.io/projected/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-kube-api-access-hts5n\") pod \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " Oct 31 16:05:39.135594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.135598 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-trusted-ca-bundle\") pod \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " Oct 31 16:05:39.135814 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.135644 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-archives-path\") pod \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " Oct 31 16:05:39.135814 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.135794 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-service-ca-bundle\") pod \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\" (UID: \"fa967dc1-1a67-4042-ab72-7eecf73eb5ee\") " Oct 31 16:05:39.136070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.135989 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-archives-path" pod "fa967dc1-1a67-4042-ab72-7eecf73eb5ee" (UID: "fa967dc1-1a67-4042-ab72-7eecf73eb5ee"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:39.136204 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.136093 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-trusted-ca-bundle" pod "fa967dc1-1a67-4042-ab72-7eecf73eb5ee" (UID: "fa967dc1-1a67-4042-ab72-7eecf73eb5ee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:39.136204 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.136124 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-service-ca-bundle" pod "fa967dc1-1a67-4042-ab72-7eecf73eb5ee" (UID: "fa967dc1-1a67-4042-ab72-7eecf73eb5ee"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:39.137841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.137814 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-kube-api-access-hts5n" pod "fa967dc1-1a67-4042-ab72-7eecf73eb5ee" (UID: "fa967dc1-1a67-4042-ab72-7eecf73eb5ee"). InnerVolumeSpecName "kube-api-access-hts5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:39.237270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.237221 4880 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-service-ca-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:05:39.237270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.237262 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hts5n\" (UniqueName: \"kubernetes.io/projected/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-kube-api-access-hts5n\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:05:39.237270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.237273 4880 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-trusted-ca-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:05:39.237270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.237284 4880 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fa967dc1-1a67-4042-ab72-7eecf73eb5ee-archives-path\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:05:39.867930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.867906 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" Oct 31 16:05:39.868102 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.867902 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hlqrl-22v6k" event={"ID":"fa967dc1-1a67-4042-ab72-7eecf73eb5ee","Type":"ContainerDied","Data":"4ee538e8782e916d99a87faa20afc2d06e0c565cdc1bd133de1b68c018b36976"} Oct 31 16:05:39.868102 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:39.868028 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ee538e8782e916d99a87faa20afc2d06e0c565cdc1bd133de1b68c018b36976" Oct 31 16:05:50.170919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:50.170886 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:50.409979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:50.409943 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:51.088090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.088062 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2"] Oct 31 16:05:51.088359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.088347 4880 cpu_manager.go:414] "containerMap: removing container" podUID="fa967dc1-1a67-4042-ab72-7eecf73eb5ee" containerName="insights-gathering" Oct 31 16:05:51.088402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.088360 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa967dc1-1a67-4042-ab72-7eecf73eb5ee" containerName="insights-gathering" Oct 31 16:05:51.088434 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.088427 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa967dc1-1a67-4042-ab72-7eecf73eb5ee" containerName="insights-gathering" Oct 31 16:05:51.090358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.090342 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.114499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.114478 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.114612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.114513 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v64p\" (UniqueName: \"kubernetes.io/projected/f47ea4aa-bcbe-4818-a7c5-de107988e30a-kube-api-access-8v64p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.114612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.114538 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.189339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.189311 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-z24xd\"" Oct 31 16:05:51.189339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.189327 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 16:05:51.189706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.189411 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 16:05:51.190472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.190455 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2"] Oct 31 16:05:51.214936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.214911 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.215058 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.214948 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v64p\" (UniqueName: \"kubernetes.io/projected/f47ea4aa-bcbe-4818-a7c5-de107988e30a-kube-api-access-8v64p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.215058 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.214974 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.215309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.215293 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.215347 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.215337 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.268336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.268311 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v64p\" (UniqueName: \"kubernetes.io/projected/f47ea4aa-bcbe-4818-a7c5-de107988e30a-kube-api-access-8v64p\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.402754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.402730 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:05:51.488679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.488636 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:51.648039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.648005 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:51.764344 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.764275 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2"] Oct 31 16:05:51.767816 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:51.767783 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47ea4aa_bcbe_4818_a7c5_de107988e30a.slice/crio-31434971b695fc72bfb136bd72e4cecf7f7d91c87bfc27076341ef06e6f349aa WatchSource:0}: Error finding container 31434971b695fc72bfb136bd72e4cecf7f7d91c87bfc27076341ef06e6f349aa: Status 404 returned error can't find the container with id 31434971b695fc72bfb136bd72e4cecf7f7d91c87bfc27076341ef06e6f349aa Oct 31 16:05:51.911811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:51.911781 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" event={"ID":"f47ea4aa-bcbe-4818-a7c5-de107988e30a","Type":"ContainerStarted","Data":"31434971b695fc72bfb136bd72e4cecf7f7d91c87bfc27076341ef06e6f349aa"} Oct 31 16:05:52.042448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.042390 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb"] Oct 31 16:05:52.044916 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.044902 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.093300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.093277 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-fhlfh\"" Oct 31 16:05:52.098613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.098591 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb"] Oct 31 16:05:52.121966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.121942 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.122075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.121984 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.122075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.122013 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6jj\" (UniqueName: \"kubernetes.io/projected/700ad351-7585-433f-9931-8ba86340af4d-kube-api-access-bq6jj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.138388 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.138362 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg"] Oct 31 16:05:52.140384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.140367 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.188386 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.188364 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-glbvd\"" Oct 31 16:05:52.194314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.194283 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg"] Oct 31 16:05:52.222671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.222646 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.222807 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.222683 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6jj\" (UniqueName: \"kubernetes.io/projected/700ad351-7585-433f-9931-8ba86340af4d-kube-api-access-bq6jj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.222807 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.222762 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.222930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.222869 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.222930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.222896 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.222930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.222925 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-857j7\" (UniqueName: \"kubernetes.io/projected/66b944ef-1b01-4c39-965f-62c61f3ac460-kube-api-access-857j7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.223085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.222996 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.223085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.223063 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.237040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.237011 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm"] Oct 31 16:05:52.239339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.239322 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.275334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.275308 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6jj\" (UniqueName: \"kubernetes.io/projected/700ad351-7585-433f-9931-8ba86340af4d-kube-api-access-bq6jj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.287607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.287587 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vlwr7\"" Oct 31 16:05:52.293559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.293500 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm"] Oct 31 16:05:52.323762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.323735 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.323762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.323763 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.323966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.323788 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-857j7\" (UniqueName: \"kubernetes.io/projected/66b944ef-1b01-4c39-965f-62c61f3ac460-kube-api-access-857j7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.323966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.323824 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.323966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.323852 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.323966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.323908 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxlm\" (UniqueName: \"kubernetes.io/projected/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-kube-api-access-5mxlm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.324211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.324196 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.324255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.324236 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.335416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.335388 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65"] Oct 31 16:05:52.337638 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.337621 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.357623 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.357598 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:05:52.379843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.379819 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-857j7\" (UniqueName: \"kubernetes.io/projected/66b944ef-1b01-4c39-965f-62c61f3ac460-kube-api-access-857j7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.388035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.388010 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-g2m7t\"" Oct 31 16:05:52.418365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.418337 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65"] Oct 31 16:05:52.425091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425059 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.425237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425125 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wwqr\" (UniqueName: \"kubernetes.io/projected/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-kube-api-access-8wwqr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.425295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425272 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.425354 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425339 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.425400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425371 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.425448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425398 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxlm\" (UniqueName: \"kubernetes.io/projected/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-kube-api-access-5mxlm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.425735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425713 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.426492 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.425729 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.443615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.443581 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:52.452366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.452348 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:05:52.478273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.478249 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxlm\" (UniqueName: \"kubernetes.io/projected/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-kube-api-access-5mxlm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.526217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.526191 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.526358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.526235 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wwqr\" (UniqueName: \"kubernetes.io/projected/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-kube-api-access-8wwqr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.526358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.526310 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.526606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.526589 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.526666 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.526608 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.551337 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.551280 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:05:52.579961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.579931 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wwqr\" (UniqueName: \"kubernetes.io/projected/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-kube-api-access-8wwqr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.649430 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.649400 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:05:52.721777 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.721748 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb"] Oct 31 16:05:52.724832 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:52.724802 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700ad351_7585_433f_9931_8ba86340af4d.slice/crio-b8f928ceaf602adf83947ad3768c9207db1e25cc524bec37df0f3ced62e64c02 WatchSource:0}: Error finding container b8f928ceaf602adf83947ad3768c9207db1e25cc524bec37df0f3ced62e64c02: Status 404 returned error can't find the container with id b8f928ceaf602adf83947ad3768c9207db1e25cc524bec37df0f3ced62e64c02 Oct 31 16:05:52.816593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.816469 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg"] Oct 31 16:05:52.819081 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:52.819054 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b944ef_1b01_4c39_965f_62c61f3ac460.slice/crio-fba2ff03600b8223bfcfda04621d96435d41bb8ade15eb918ebca7ac83497922 WatchSource:0}: Error finding container fba2ff03600b8223bfcfda04621d96435d41bb8ade15eb918ebca7ac83497922: Status 404 returned error can't find the container with id fba2ff03600b8223bfcfda04621d96435d41bb8ade15eb918ebca7ac83497922 Oct 31 16:05:52.909385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.909354 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm"] Oct 31 16:05:52.912446 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:52.912422 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd2c90f_11c8_46ce_bc99_7bcc746f69f6.slice/crio-c4956fec8f57471c895463fc0df457a2e7930078dcf3b58c4d79e1b410baa709 WatchSource:0}: Error finding container c4956fec8f57471c895463fc0df457a2e7930078dcf3b58c4d79e1b410baa709: Status 404 returned error can't find the container with id c4956fec8f57471c895463fc0df457a2e7930078dcf3b58c4d79e1b410baa709 Oct 31 16:05:52.917129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.917088 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" event={"ID":"66b944ef-1b01-4c39-965f-62c61f3ac460","Type":"ContainerStarted","Data":"fba2ff03600b8223bfcfda04621d96435d41bb8ade15eb918ebca7ac83497922"} Oct 31 16:05:52.918149 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:52.918117 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" event={"ID":"700ad351-7585-433f-9931-8ba86340af4d","Type":"ContainerStarted","Data":"b8f928ceaf602adf83947ad3768c9207db1e25cc524bec37df0f3ced62e64c02"} Oct 31 16:05:53.014218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:53.014185 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65"] Oct 31 16:05:53.020827 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:53.020800 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b792f2_6bb0_4da0_a3d8_84e7cf6e7082.slice/crio-e2de99650658e74767d2ea5243aefb9a2d81949b44946edc3f8014ac79263e2c WatchSource:0}: Error finding container e2de99650658e74767d2ea5243aefb9a2d81949b44946edc3f8014ac79263e2c: Status 404 returned error can't find the container with id e2de99650658e74767d2ea5243aefb9a2d81949b44946edc3f8014ac79263e2c Oct 31 16:05:53.927033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:53.926965 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" event={"ID":"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6","Type":"ContainerStarted","Data":"c4956fec8f57471c895463fc0df457a2e7930078dcf3b58c4d79e1b410baa709"} Oct 31 16:05:53.929387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:53.929352 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" event={"ID":"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082","Type":"ContainerStarted","Data":"e2de99650658e74767d2ea5243aefb9a2d81949b44946edc3f8014ac79263e2c"} Oct 31 16:05:54.325877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.323825 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v"] Oct 31 16:05:54.327501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.326854 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.341665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.341632 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.341795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.341708 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4tj\" (UniqueName: \"kubernetes.io/projected/f6494132-013c-40b9-b7c1-c3c852cee65c-kube-api-access-bx4tj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.341795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.341744 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.426496 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.426452 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v"] Oct 31 16:05:54.426717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.426652 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gj9fw\"" Oct 31 16:05:54.442837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.442803 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4tj\" (UniqueName: \"kubernetes.io/projected/f6494132-013c-40b9-b7c1-c3c852cee65c-kube-api-access-bx4tj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.442978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.442859 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.442978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.442918 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.443474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.443420 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.444009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.443984 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.509409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.509359 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4tj\" (UniqueName: \"kubernetes.io/projected/f6494132-013c-40b9-b7c1-c3c852cee65c-kube-api-access-bx4tj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.539648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.539620 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh"] Oct 31 16:05:54.542480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.542048 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.590829 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.590745 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-5n2ft\"" Oct 31 16:05:54.597986 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.597946 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh"] Oct 31 16:05:54.644873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.644662 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.644873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.644708 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.644873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.644738 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trf75\" (UniqueName: \"kubernetes.io/projected/b477a592-8028-4f37-b90f-641df89bad23-kube-api-access-trf75\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.646326 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.646253 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:05:54.752520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.751448 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.752520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.751497 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.752520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.751545 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trf75\" (UniqueName: \"kubernetes.io/projected/b477a592-8028-4f37-b90f-641df89bad23-kube-api-access-trf75\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.752520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.752472 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.753091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.753040 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.805459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.805427 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trf75\" (UniqueName: \"kubernetes.io/projected/b477a592-8028-4f37-b90f-641df89bad23-kube-api-access-trf75\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:54.859552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:54.859470 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:05:55.051799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:55.051765 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v"] Oct 31 16:05:55.057079 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:55.057040 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6494132_013c_40b9_b7c1_c3c852cee65c.slice/crio-2fc4f8e7f18f90311d7f7dd71413570702c9e2536f2aeedcfca5bc7e15b84ef5 WatchSource:0}: Error finding container 2fc4f8e7f18f90311d7f7dd71413570702c9e2536f2aeedcfca5bc7e15b84ef5: Status 404 returned error can't find the container with id 2fc4f8e7f18f90311d7f7dd71413570702c9e2536f2aeedcfca5bc7e15b84ef5 Oct 31 16:05:55.265197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:55.265161 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh"] Oct 31 16:05:55.945899 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:55.945801 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" event={"ID":"f6494132-013c-40b9-b7c1-c3c852cee65c","Type":"ContainerStarted","Data":"2fc4f8e7f18f90311d7f7dd71413570702c9e2536f2aeedcfca5bc7e15b84ef5"} Oct 31 16:05:55.948116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:55.948040 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" event={"ID":"b477a592-8028-4f37-b90f-641df89bad23","Type":"ContainerStarted","Data":"17659408e9bbc27a643dabc16dd0b320fe3bceae52933b02c424bab84b8275d9"} Oct 31 16:05:57.754179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:57.754120 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:57.961731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:57.961694 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:58.276382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.274322 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm"] Oct 31 16:05:58.277989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.277550 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.330129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.330071 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm"] Oct 31 16:05:58.385417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.385384 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.385593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.385507 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9tt9\" (UniqueName: \"kubernetes.io/projected/4219ddc1-da11-43be-abf9-d0de6302a37b-kube-api-access-s9tt9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.385593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.385569 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.487428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.486441 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9tt9\" (UniqueName: \"kubernetes.io/projected/4219ddc1-da11-43be-abf9-d0de6302a37b-kube-api-access-s9tt9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.487428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.486542 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.487428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.486621 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.487428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.487064 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.487428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.487379 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.565477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.565374 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9tt9\" (UniqueName: \"kubernetes.io/projected/4219ddc1-da11-43be-abf9-d0de6302a37b-kube-api-access-s9tt9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.566988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.566956 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6"] Oct 31 16:05:58.569667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.569648 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.600823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.600614 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:05:58.619821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.619753 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gw56j\"" Oct 31 16:05:58.620237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.620201 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 16:05:58.628993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.628963 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6"] Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687793 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687836 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687881 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22wv\" (UniqueName: \"kubernetes.io/projected/363aa6f3-aaba-4371-b953-557fc18a8d3f-kube-api-access-z22wv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687900 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687917 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363aa6f3-aaba-4371-b953-557fc18a8d3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687937 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687959 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.687982 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.688776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.688009 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.690228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.690124 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789427 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789469 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789540 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789585 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22wv\" (UniqueName: \"kubernetes.io/projected/363aa6f3-aaba-4371-b953-557fc18a8d3f-kube-api-access-z22wv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789605 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789622 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363aa6f3-aaba-4371-b953-557fc18a8d3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789663 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789683 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.790685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.789704 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.792165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.791871 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.792318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.792293 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.792440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.792415 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363aa6f3-aaba-4371-b953-557fc18a8d3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.792745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.792707 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.792839 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.792781 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.803679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.803654 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.804554 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.804534 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.851074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.851007 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl"] Oct 31 16:05:58.853638 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.853612 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363aa6f3-aaba-4371-b953-557fc18a8d3f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.854997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.854968 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.856060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.856038 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22wv\" (UniqueName: \"kubernetes.io/projected/363aa6f3-aaba-4371-b953-557fc18a8d3f-kube-api-access-z22wv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-jdhp6\" (UID: \"363aa6f3-aaba-4371-b953-557fc18a8d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.887417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.886760 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:05:58.890800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.890776 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.890894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.890842 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.890949 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.890891 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.890949 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.890919 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.891050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.890953 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.891050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.890981 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlrb\" (UniqueName: \"kubernetes.io/projected/8a5711ec-01b8-47ac-9349-d8d1184711b3-kube-api-access-mtlrb\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.891050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.891032 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.891158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.891061 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.891158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.891085 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5711ec-01b8-47ac-9349-d8d1184711b3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.903567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.903473 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:58.903798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.903780 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-s6rtm\"" Oct 31 16:05:58.909241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.909191 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl"] Oct 31 16:05:58.992790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.992754 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlrb\" (UniqueName: \"kubernetes.io/projected/8a5711ec-01b8-47ac-9349-d8d1184711b3-kube-api-access-mtlrb\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993181 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993234 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993260 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5711ec-01b8-47ac-9349-d8d1184711b3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993313 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993376 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993434 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993473 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.993658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.993511 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.995166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.995124 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.995945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.995918 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.996449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.996425 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.997079 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.997049 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5711ec-01b8-47ac-9349-d8d1184711b3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.997380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.997356 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:58.999003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:58.998960 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:59.011038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.010682 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:59.047167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.047030 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlrb\" (UniqueName: \"kubernetes.io/projected/8a5711ec-01b8-47ac-9349-d8d1184711b3-kube-api-access-mtlrb\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:59.050745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.050687 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5711ec-01b8-47ac-9349-d8d1184711b3-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl\" (UID: \"8a5711ec-01b8-47ac-9349-d8d1184711b3\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:59.063995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.063819 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:05:59.080834 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.079943 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm"] Oct 31 16:05:59.095182 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:59.095156 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4219ddc1_da11_43be_abf9_d0de6302a37b.slice/crio-4e78e9e4395ed1431be5db514e1d6544809e26c13a1ab6d1b365096cefa8b146 WatchSource:0}: Error finding container 4e78e9e4395ed1431be5db514e1d6544809e26c13a1ab6d1b365096cefa8b146: Status 404 returned error can't find the container with id 4e78e9e4395ed1431be5db514e1d6544809e26c13a1ab6d1b365096cefa8b146 Oct 31 16:05:59.174124 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.174097 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:05:59.289960 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.289923 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w"] Oct 31 16:05:59.297318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.296433 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.314160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.312454 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z"] Oct 31 16:05:59.316015 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.315782 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.334806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.334776 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6"] Oct 31 16:05:59.341594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.341574 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs"] Oct 31 16:05:59.345781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.345761 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.348360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.348337 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w"] Oct 31 16:05:59.349633 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:59.349606 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363aa6f3_aaba_4371_b953_557fc18a8d3f.slice/crio-a4028700e3509f27fce21ab8e69aac91909304a9adb671bd7f26bc9eaa5bdef0 WatchSource:0}: Error finding container a4028700e3509f27fce21ab8e69aac91909304a9adb671bd7f26bc9eaa5bdef0: Status 404 returned error can't find the container with id a4028700e3509f27fce21ab8e69aac91909304a9adb671bd7f26bc9eaa5bdef0 Oct 31 16:05:59.372808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.372583 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs"] Oct 31 16:05:59.379908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.379881 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z"] Oct 31 16:05:59.380044 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.380029 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.396617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.396601 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs"] Oct 31 16:05:59.398989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.398966 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.399113 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399007 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbgn\" (UniqueName: \"kubernetes.io/projected/9938c0be-e9db-4358-8118-725a458b7a18-kube-api-access-hhbgn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.399113 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399097 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.399243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399160 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbdf\" (UniqueName: \"kubernetes.io/projected/ff6b4072-258a-4a64-b900-141c82967bb6-kube-api-access-lsbdf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.399243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399195 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.399335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399246 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.399335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399280 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.399636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399612 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.399781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399753 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkdc\" (UniqueName: \"kubernetes.io/projected/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-kube-api-access-kgkdc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.399846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399786 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.399846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399814 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.400193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.399848 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85znq\" (UniqueName: \"kubernetes.io/projected/874393ca-7c09-42ce-8149-d778433b0ea6-kube-api-access-85znq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.425014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.424949 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs"] Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501057 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501150 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501185 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501223 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501269 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkdc\" (UniqueName: \"kubernetes.io/projected/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-kube-api-access-kgkdc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501298 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501331 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501362 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85znq\" (UniqueName: \"kubernetes.io/projected/874393ca-7c09-42ce-8149-d778433b0ea6-kube-api-access-85znq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501402 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501429 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbgn\" (UniqueName: \"kubernetes.io/projected/9938c0be-e9db-4358-8118-725a458b7a18-kube-api-access-hhbgn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501460 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.501505 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbdf\" (UniqueName: \"kubernetes.io/projected/ff6b4072-258a-4a64-b900-141c82967bb6-kube-api-access-lsbdf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.502228 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.502652 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.502910 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.503322 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.503775 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.504062 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.505162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.504644 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.507727 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.507659 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.555562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.555527 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbdf\" (UniqueName: \"kubernetes.io/projected/ff6b4072-258a-4a64-b900-141c82967bb6-kube-api-access-lsbdf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.557425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.557387 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85znq\" (UniqueName: \"kubernetes.io/projected/874393ca-7c09-42ce-8149-d778433b0ea6-kube-api-access-85znq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:05:59.557522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.557431 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkdc\" (UniqueName: \"kubernetes.io/projected/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-kube-api-access-kgkdc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.559883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.559864 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbgn\" (UniqueName: \"kubernetes.io/projected/9938c0be-e9db-4358-8118-725a458b7a18-kube-api-access-hhbgn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.605083 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.601259 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl"] Oct 31 16:05:59.610195 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:05:59.610159 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5711ec_01b8_47ac_9349_d8d1184711b3.slice/crio-96bea6e1b6219ca3ceae9390c753b017c048cd51b2de49bf6bafdfaab9ecd658 WatchSource:0}: Error finding container 96bea6e1b6219ca3ceae9390c753b017c048cd51b2de49bf6bafdfaab9ecd658: Status 404 returned error can't find the container with id 96bea6e1b6219ca3ceae9390c753b017c048cd51b2de49bf6bafdfaab9ecd658 Oct 31 16:05:59.615300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.615279 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:05:59.644908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.644882 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:05:59.677129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.676791 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:05:59.705291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:05:59.704099 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:06:00.014103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:00.014022 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" event={"ID":"8a5711ec-01b8-47ac-9349-d8d1184711b3","Type":"ContainerStarted","Data":"96bea6e1b6219ca3ceae9390c753b017c048cd51b2de49bf6bafdfaab9ecd658"} Oct 31 16:06:00.020074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:00.019911 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w"] Oct 31 16:06:00.022547 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:06:00.022513 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551d37f5_a4f2_45ae_b26e_c6a35fdcc5b5.slice/crio-9e9c584f9bf600fe1afedcc0690db450c5c8c353aa91c445d8ad3b525c44551d WatchSource:0}: Error finding container 9e9c584f9bf600fe1afedcc0690db450c5c8c353aa91c445d8ad3b525c44551d: Status 404 returned error can't find the container with id 9e9c584f9bf600fe1afedcc0690db450c5c8c353aa91c445d8ad3b525c44551d Oct 31 16:06:00.023227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:00.023122 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" event={"ID":"363aa6f3-aaba-4371-b953-557fc18a8d3f","Type":"ContainerStarted","Data":"a4028700e3509f27fce21ab8e69aac91909304a9adb671bd7f26bc9eaa5bdef0"} Oct 31 16:06:00.025458 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:00.025317 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" event={"ID":"4219ddc1-da11-43be-abf9-d0de6302a37b","Type":"ContainerStarted","Data":"4e78e9e4395ed1431be5db514e1d6544809e26c13a1ab6d1b365096cefa8b146"} Oct 31 16:06:00.052778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:00.052013 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z"] Oct 31 16:06:00.059428 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:06:00.059391 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9938c0be_e9db_4358_8118_725a458b7a18.slice/crio-60e7ddc9508791b0a80c5c886b3597d1df5bac73151e20bac708ec4a909b0989 WatchSource:0}: Error finding container 60e7ddc9508791b0a80c5c886b3597d1df5bac73151e20bac708ec4a909b0989: Status 404 returned error can't find the container with id 60e7ddc9508791b0a80c5c886b3597d1df5bac73151e20bac708ec4a909b0989 Oct 31 16:06:00.120837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:00.120773 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs"] Oct 31 16:06:00.124208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:00.123845 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs"] Oct 31 16:06:00.134917 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:06:00.134874 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6b4072_258a_4a64_b900_141c82967bb6.slice/crio-56f75b9bb47f4e822ebfa7554ab0d4efb7bdf5027165a45290a0dd125ef2e482 WatchSource:0}: Error finding container 56f75b9bb47f4e822ebfa7554ab0d4efb7bdf5027165a45290a0dd125ef2e482: Status 404 returned error can't find the container with id 56f75b9bb47f4e822ebfa7554ab0d4efb7bdf5027165a45290a0dd125ef2e482 Oct 31 16:06:00.142878 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:06:00.142851 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874393ca_7c09_42ce_8149_d778433b0ea6.slice/crio-b05f680d8083bb17ecee164357006a89275c3f7ff0001c04bd8ebe6a5e743302 WatchSource:0}: Error finding container b05f680d8083bb17ecee164357006a89275c3f7ff0001c04bd8ebe6a5e743302: Status 404 returned error can't find the container with id b05f680d8083bb17ecee164357006a89275c3f7ff0001c04bd8ebe6a5e743302 Oct 31 16:06:01.042160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:01.042083 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" event={"ID":"874393ca-7c09-42ce-8149-d778433b0ea6","Type":"ContainerStarted","Data":"b05f680d8083bb17ecee164357006a89275c3f7ff0001c04bd8ebe6a5e743302"} Oct 31 16:06:01.045244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:01.044650 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" event={"ID":"9938c0be-e9db-4358-8118-725a458b7a18","Type":"ContainerStarted","Data":"60e7ddc9508791b0a80c5c886b3597d1df5bac73151e20bac708ec4a909b0989"} Oct 31 16:06:01.054802 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:01.054769 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" event={"ID":"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5","Type":"ContainerStarted","Data":"9e9c584f9bf600fe1afedcc0690db450c5c8c353aa91c445d8ad3b525c44551d"} Oct 31 16:06:01.056177 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:01.056057 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" event={"ID":"ff6b4072-258a-4a64-b900-141c82967bb6","Type":"ContainerStarted","Data":"56f75b9bb47f4e822ebfa7554ab0d4efb7bdf5027165a45290a0dd125ef2e482"} Oct 31 16:06:02.944577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:02.944539 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:03.920166 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:06:03.917828 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:06:11.876185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:11.876150 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:13.235480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:13.235441 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" event={"ID":"f47ea4aa-bcbe-4818-a7c5-de107988e30a","Type":"ContainerStarted","Data":"3f5d8381f6100bc56891d7fb5eea0dcab197628448d32387de0c928af0308415"} Oct 31 16:06:13.302415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:13.301718 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:06:13.302415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:13.301811 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:06:13.302415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:13.301861 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:06:13.501367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:13.501324 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:06:13.501487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:13.501432 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:06:13.501487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:13.501476 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:06:14.251570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.250534 4880 generic.go:356] "Generic (PLEG): container finished" podID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerID="3f5d8381f6100bc56891d7fb5eea0dcab197628448d32387de0c928af0308415" exitCode=0 Oct 31 16:06:14.251570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.250626 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" event={"ID":"f47ea4aa-bcbe-4818-a7c5-de107988e30a","Type":"ContainerDied","Data":"3f5d8381f6100bc56891d7fb5eea0dcab197628448d32387de0c928af0308415"} Oct 31 16:06:14.257178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.256579 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" event={"ID":"8a5711ec-01b8-47ac-9349-d8d1184711b3","Type":"ContainerStarted","Data":"601e83c6c732afa58018ef63988f9e73a7362502cb299116d3552a56036f9856"} Oct 31 16:06:14.260822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.260303 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" event={"ID":"363aa6f3-aaba-4371-b953-557fc18a8d3f","Type":"ContainerStarted","Data":"6517f3ecb76d507381ad3ecba39953738e6591345f5ce5d3c1348b25c2d0b19e"} Oct 31 16:06:14.468891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.468038 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podStartSLOduration=2.529531785 podStartE2EDuration="16.468018606s" totalImagesPullingTime="13.938486821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:14.465786783 +0000 UTC m=+221.501081587" watchObservedRunningTime="2026-10-31 16:06:14.468018606 +0000 UTC m=+221.503313412" Oct 31 16:06:14.475403 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.475364 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:14.578129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.577281 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" podStartSLOduration=2.692091699 podStartE2EDuration="16.577258686s" totalImagesPullingTime="13.885166987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:14.576167127 +0000 UTC m=+221.611461932" watchObservedRunningTime="2026-10-31 16:06:14.577258686 +0000 UTC m=+221.612553492" Oct 31 16:06:14.887552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.887254 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:06:14.893380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:14.893305 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:06:15.175715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.175690 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:06:15.181294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.181269 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:06:15.269169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.269109 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" event={"ID":"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6","Type":"ContainerStarted","Data":"9f880c33f12cb6f8edbe3e62cd6e298954cf7376e6b791891ff0797c8442814d"} Oct 31 16:06:15.273193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.271795 4880 generic.go:356] "Generic (PLEG): container finished" podID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerID="f207c4f012b6d0a4df08bf2f4db6c881e7b4efa41d98f59439a932c248810290" exitCode=0 Oct 31 16:06:15.273193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.271901 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" event={"ID":"4219ddc1-da11-43be-abf9-d0de6302a37b","Type":"ContainerDied","Data":"f207c4f012b6d0a4df08bf2f4db6c881e7b4efa41d98f59439a932c248810290"} Oct 31 16:06:15.275527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.275502 4880 generic.go:356] "Generic (PLEG): container finished" podID="9938c0be-e9db-4358-8118-725a458b7a18" containerID="80927b67591be3170c574b0bbcaaa427dfad170b5b3e3913afdff9a261906b69" exitCode=0 Oct 31 16:06:15.276652 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.276629 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" event={"ID":"9938c0be-e9db-4358-8118-725a458b7a18","Type":"ContainerDied","Data":"80927b67591be3170c574b0bbcaaa427dfad170b5b3e3913afdff9a261906b69"} Oct 31 16:06:15.276760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.276674 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:06:15.277430 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.277385 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:06:15.278126 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.278106 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:06:15.278255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:15.278233 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl" Oct 31 16:06:16.281007 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.280943 4880 generic.go:356] "Generic (PLEG): container finished" podID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerID="9f880c33f12cb6f8edbe3e62cd6e298954cf7376e6b791891ff0797c8442814d" exitCode=0 Oct 31 16:06:16.281453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.281019 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" event={"ID":"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6","Type":"ContainerDied","Data":"9f880c33f12cb6f8edbe3e62cd6e298954cf7376e6b791891ff0797c8442814d"} Oct 31 16:06:16.282591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.282569 4880 generic.go:356] "Generic (PLEG): container finished" podID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerID="c8ba4f70630304fb3a0bc406d4ce3eed963addcb9b76485a84aed87bbd6c61f0" exitCode=0 Oct 31 16:06:16.282692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.282643 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" event={"ID":"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082","Type":"ContainerDied","Data":"c8ba4f70630304fb3a0bc406d4ce3eed963addcb9b76485a84aed87bbd6c61f0"} Oct 31 16:06:16.284051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.284032 4880 generic.go:356] "Generic (PLEG): container finished" podID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerID="cae7051662e8133e9022ff8b64db97f11e12cdae63126cfea6a09057e7b4a4b2" exitCode=0 Oct 31 16:06:16.284115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.284088 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" event={"ID":"66b944ef-1b01-4c39-965f-62c61f3ac460","Type":"ContainerDied","Data":"cae7051662e8133e9022ff8b64db97f11e12cdae63126cfea6a09057e7b4a4b2"} Oct 31 16:06:16.285564 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.285544 4880 generic.go:356] "Generic (PLEG): container finished" podID="ff6b4072-258a-4a64-b900-141c82967bb6" containerID="ebd2e7609cc1435a55ae8a675289d7a5bc85b32eb2d02e7f42405b32c72a9657" exitCode=0 Oct 31 16:06:16.285640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:16.285602 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" event={"ID":"ff6b4072-258a-4a64-b900-141c82967bb6","Type":"ContainerDied","Data":"ebd2e7609cc1435a55ae8a675289d7a5bc85b32eb2d02e7f42405b32c72a9657"} Oct 31 16:06:17.291516 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:17.291488 4880 generic.go:356] "Generic (PLEG): container finished" podID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerID="ebdf70b939d840bc498d90fc81e5889141cb182e0f983f49952bbda086857cee" exitCode=0 Oct 31 16:06:17.291920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:17.291576 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" event={"ID":"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5","Type":"ContainerDied","Data":"ebdf70b939d840bc498d90fc81e5889141cb182e0f983f49952bbda086857cee"} Oct 31 16:06:17.293378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:17.293274 4880 generic.go:356] "Generic (PLEG): container finished" podID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerID="98d0cfe092983070e4ddf612f3c41db05670b81953c8e28b56696c6541db73db" exitCode=0 Oct 31 16:06:17.293378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:17.293305 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" event={"ID":"f6494132-013c-40b9-b7c1-c3c852cee65c","Type":"ContainerDied","Data":"98d0cfe092983070e4ddf612f3c41db05670b81953c8e28b56696c6541db73db"} Oct 31 16:06:17.294999 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:17.294974 4880 generic.go:356] "Generic (PLEG): container finished" podID="b477a592-8028-4f37-b90f-641df89bad23" containerID="61f069220d4fa29f8aa3992da96df8443c4dec8cdd5c0a41b31b4028bb28df18" exitCode=0 Oct 31 16:06:17.295180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:17.295085 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" event={"ID":"b477a592-8028-4f37-b90f-641df89bad23","Type":"ContainerDied","Data":"61f069220d4fa29f8aa3992da96df8443c4dec8cdd5c0a41b31b4028bb28df18"} Oct 31 16:06:18.301346 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.301314 4880 generic.go:356] "Generic (PLEG): container finished" podID="700ad351-7585-433f-9931-8ba86340af4d" containerID="39e04b43a0d789b02ee0f0cae526c6f35819a61468078d7a8b2f63c5cf7d092e" exitCode=0 Oct 31 16:06:18.301740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.301389 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" event={"ID":"700ad351-7585-433f-9931-8ba86340af4d","Type":"ContainerDied","Data":"39e04b43a0d789b02ee0f0cae526c6f35819a61468078d7a8b2f63c5cf7d092e"} Oct 31 16:06:18.303224 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.303204 4880 generic.go:356] "Generic (PLEG): container finished" podID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerID="9cc45783d21d3a67ee8c4bf714292e4f2be4e2311bb14cfb87c41739233c68bd" exitCode=0 Oct 31 16:06:18.303337 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.303284 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" event={"ID":"f47ea4aa-bcbe-4818-a7c5-de107988e30a","Type":"ContainerDied","Data":"9cc45783d21d3a67ee8c4bf714292e4f2be4e2311bb14cfb87c41739233c68bd"} Oct 31 16:06:18.304946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.304926 4880 generic.go:356] "Generic (PLEG): container finished" podID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerID="08ab40b7c57aa4b9c6a5345c7ca3b61e982609a60a5ce630dd8d80b0ce94ebb7" exitCode=0 Oct 31 16:06:18.305039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.304988 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" event={"ID":"4219ddc1-da11-43be-abf9-d0de6302a37b","Type":"ContainerDied","Data":"08ab40b7c57aa4b9c6a5345c7ca3b61e982609a60a5ce630dd8d80b0ce94ebb7"} Oct 31 16:06:18.307588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.307311 4880 generic.go:356] "Generic (PLEG): container finished" podID="874393ca-7c09-42ce-8149-d778433b0ea6" containerID="bd3e29e5497de7029638871512c6f6b97e3b4bb2783c76734cf8917cc8358680" exitCode=0 Oct 31 16:06:18.307588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:18.307393 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" event={"ID":"874393ca-7c09-42ce-8149-d778433b0ea6","Type":"ContainerDied","Data":"bd3e29e5497de7029638871512c6f6b97e3b4bb2783c76734cf8917cc8358680"} Oct 31 16:06:19.313665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:19.313627 4880 generic.go:356] "Generic (PLEG): container finished" podID="ff6b4072-258a-4a64-b900-141c82967bb6" containerID="ea5f931bdbfd7815fe48c969a590bb026b353dc249b0a949cb3289a8eb7ae43d" exitCode=0 Oct 31 16:06:19.314090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:19.313707 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" event={"ID":"ff6b4072-258a-4a64-b900-141c82967bb6","Type":"ContainerDied","Data":"ea5f931bdbfd7815fe48c969a590bb026b353dc249b0a949cb3289a8eb7ae43d"} Oct 31 16:06:19.315720 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:19.315694 4880 generic.go:356] "Generic (PLEG): container finished" podID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerID="7d0f48e644762ca3070b3caa3a8b73d69349f313dcb6b2790d072f82f7f4fd4b" exitCode=0 Oct 31 16:06:19.315841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:19.315747 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" event={"ID":"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082","Type":"ContainerDied","Data":"7d0f48e644762ca3070b3caa3a8b73d69349f313dcb6b2790d072f82f7f4fd4b"} Oct 31 16:06:19.317884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:19.317861 4880 generic.go:356] "Generic (PLEG): container finished" podID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerID="d6c67682c70c14ee18cb21b086a4af8c15fbd1458026f48280c6b60f47d8f556" exitCode=0 Oct 31 16:06:19.317941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:19.317900 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" event={"ID":"66b944ef-1b01-4c39-965f-62c61f3ac460","Type":"ContainerDied","Data":"d6c67682c70c14ee18cb21b086a4af8c15fbd1458026f48280c6b60f47d8f556"} Oct 31 16:06:20.327199 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:20.327166 4880 generic.go:356] "Generic (PLEG): container finished" podID="700ad351-7585-433f-9931-8ba86340af4d" containerID="8bda8a3ac55abeb9c2756962dcaef1b97cdc875fa6831c07e74d07449dfa2dbc" exitCode=0 Oct 31 16:06:20.327601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:20.327319 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" event={"ID":"700ad351-7585-433f-9931-8ba86340af4d","Type":"ContainerDied","Data":"8bda8a3ac55abeb9c2756962dcaef1b97cdc875fa6831c07e74d07449dfa2dbc"} Oct 31 16:06:20.332598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:20.332572 4880 generic.go:356] "Generic (PLEG): container finished" podID="874393ca-7c09-42ce-8149-d778433b0ea6" containerID="8ed098a4aa9b30c9b306f75dd49924b4e803b5ea892683a91667733896435344" exitCode=0 Oct 31 16:06:20.333586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:20.333489 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" event={"ID":"874393ca-7c09-42ce-8149-d778433b0ea6","Type":"ContainerDied","Data":"8ed098a4aa9b30c9b306f75dd49924b4e803b5ea892683a91667733896435344"} Oct 31 16:06:21.345410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:21.345365 4880 generic.go:356] "Generic (PLEG): container finished" podID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerID="f27e02928e3c6e3f35b9bea19c962adf460d9107e6898b2921434c0cdff984a8" exitCode=0 Oct 31 16:06:21.345873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:21.345532 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" event={"ID":"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6","Type":"ContainerDied","Data":"f27e02928e3c6e3f35b9bea19c962adf460d9107e6898b2921434c0cdff984a8"} Oct 31 16:06:21.371677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:21.371646 4880 generic.go:356] "Generic (PLEG): container finished" podID="9938c0be-e9db-4358-8118-725a458b7a18" containerID="21576c049222b7708dcb513334e0abb6d82d13fcfdbc0109b91f1f639a3e8924" exitCode=0 Oct 31 16:06:21.371814 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:21.371787 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" event={"ID":"9938c0be-e9db-4358-8118-725a458b7a18","Type":"ContainerDied","Data":"21576c049222b7708dcb513334e0abb6d82d13fcfdbc0109b91f1f639a3e8924"} Oct 31 16:06:21.380390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:21.380336 4880 generic.go:356] "Generic (PLEG): container finished" podID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerID="395caf7f74a84a8e8b32b91d43517b81ae1fef93cada79a1375db4c985428cf8" exitCode=0 Oct 31 16:06:21.381165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:21.380547 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" event={"ID":"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5","Type":"ContainerDied","Data":"395caf7f74a84a8e8b32b91d43517b81ae1fef93cada79a1375db4c985428cf8"} Oct 31 16:06:24.433009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:24.432978 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:24.485707 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:24.485642 4880 generic.go:356] "Generic (PLEG): container finished" podID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerID="7c3518936266fd2acd4a38c0645ca8d411745e46b5f19d303913198851e109c4" exitCode=0 Oct 31 16:06:24.485870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:24.485738 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" event={"ID":"f6494132-013c-40b9-b7c1-c3c852cee65c","Type":"ContainerDied","Data":"7c3518936266fd2acd4a38c0645ca8d411745e46b5f19d303913198851e109c4"} Oct 31 16:06:27.540446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:27.540412 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" event={"ID":"b477a592-8028-4f37-b90f-641df89bad23","Type":"ContainerStarted","Data":"285b031c0bb68d11b2dbae45301c9e632b98ddda566e24afbbf46419e18ad9d7"} Oct 31 16:06:28.564254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:28.564205 4880 generic.go:356] "Generic (PLEG): container finished" podID="b477a592-8028-4f37-b90f-641df89bad23" containerID="285b031c0bb68d11b2dbae45301c9e632b98ddda566e24afbbf46419e18ad9d7" exitCode=0 Oct 31 16:06:28.564728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:28.564330 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" event={"ID":"b477a592-8028-4f37-b90f-641df89bad23","Type":"ContainerDied","Data":"285b031c0bb68d11b2dbae45301c9e632b98ddda566e24afbbf46419e18ad9d7"} Oct 31 16:06:33.937820 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:06:33.937746 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:06:34.746282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:34.746130 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:41.787392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:41.786510 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" event={"ID":"4219ddc1-da11-43be-abf9-d0de6302a37b","Type":"ContainerStarted","Data":"30fbd214f1bfc2185a21abb1734364351dd555932e689d2f03fe4017d1b3d867"} Oct 31 16:06:41.895057 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:41.894988 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" podStartSLOduration=3.466075701 podStartE2EDuration="43.894968521s" totalImagesPullingTime="40.42889282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:41.893220323 +0000 UTC m=+248.928515127" watchObservedRunningTime="2026-10-31 16:06:41.894968521 +0000 UTC m=+248.930263326" Oct 31 16:06:42.805151 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:42.805039 4880 generic.go:356] "Generic (PLEG): container finished" podID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerID="30fbd214f1bfc2185a21abb1734364351dd555932e689d2f03fe4017d1b3d867" exitCode=0 Oct 31 16:06:42.805617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:42.805244 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" event={"ID":"4219ddc1-da11-43be-abf9-d0de6302a37b","Type":"ContainerDied","Data":"30fbd214f1bfc2185a21abb1734364351dd555932e689d2f03fe4017d1b3d867"} Oct 31 16:06:42.815090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:42.814829 4880 generic.go:356] "Generic (PLEG): container finished" podID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerID="114036ee911844018029a365c39bc25df59a1b3dea61b92908cf04a86cd1492d" exitCode=0 Oct 31 16:06:42.815090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:42.814903 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" event={"ID":"f47ea4aa-bcbe-4818-a7c5-de107988e30a","Type":"ContainerDied","Data":"114036ee911844018029a365c39bc25df59a1b3dea61b92908cf04a86cd1492d"} Oct 31 16:06:43.830391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:43.830351 4880 generic.go:356] "Generic (PLEG): container finished" podID="ff6b4072-258a-4a64-b900-141c82967bb6" containerID="b2a90b4ed849e614e6be78b0309f03c0ac32214b188e148648aa00f1276c0f8e" exitCode=0 Oct 31 16:06:43.830872 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:43.830484 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" event={"ID":"ff6b4072-258a-4a64-b900-141c82967bb6","Type":"ContainerDied","Data":"b2a90b4ed849e614e6be78b0309f03c0ac32214b188e148648aa00f1276c0f8e"} Oct 31 16:06:43.834043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:43.834010 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" event={"ID":"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082","Type":"ContainerStarted","Data":"e9cf735949dba28db6624dac0634d015a4b3956ba49afa1a55931ae9d6380e2f"} Oct 31 16:06:43.837080 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:43.836971 4880 generic.go:356] "Generic (PLEG): container finished" podID="b477a592-8028-4f37-b90f-641df89bad23" containerID="326ae344374a47add1151550764b7f9856d0ecba45f11b75a24f4a1e806356de" exitCode=0 Oct 31 16:06:43.837332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:43.837262 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" event={"ID":"b477a592-8028-4f37-b90f-641df89bad23","Type":"ContainerDied","Data":"326ae344374a47add1151550764b7f9856d0ecba45f11b75a24f4a1e806356de"} Oct 31 16:06:44.041452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.041425 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:06:44.069098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.069074 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:06:44.131319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.131289 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-util\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-util\") pod \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " Oct 31 16:06:44.131414 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.131367 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-bundle\") pod \"4219ddc1-da11-43be-abf9-d0de6302a37b\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " Oct 31 16:06:44.131452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.131426 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4219ddc1-da11-43be-abf9-d0de6302a37b-kube-api-access-s9tt9\" (UniqueName: \"kubernetes.io/projected/4219ddc1-da11-43be-abf9-d0de6302a37b-kube-api-access-s9tt9\") pod \"4219ddc1-da11-43be-abf9-d0de6302a37b\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " Oct 31 16:06:44.131500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.131478 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-bundle\") pod \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " Oct 31 16:06:44.131544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.131511 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-util\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-util\") pod \"4219ddc1-da11-43be-abf9-d0de6302a37b\" (UID: \"4219ddc1-da11-43be-abf9-d0de6302a37b\") " Oct 31 16:06:44.131578 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.131542 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47ea4aa-bcbe-4818-a7c5-de107988e30a-kube-api-access-8v64p\" (UniqueName: \"kubernetes.io/projected/f47ea4aa-bcbe-4818-a7c5-de107988e30a-kube-api-access-8v64p\") pod \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\" (UID: \"f47ea4aa-bcbe-4818-a7c5-de107988e30a\") " Oct 31 16:06:44.132992 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.132947 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-bundle" pod "f47ea4aa-bcbe-4818-a7c5-de107988e30a" (UID: "f47ea4aa-bcbe-4818-a7c5-de107988e30a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:44.134404 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.134011 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-bundle" pod "4219ddc1-da11-43be-abf9-d0de6302a37b" (UID: "4219ddc1-da11-43be-abf9-d0de6302a37b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:44.135302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.135072 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47ea4aa-bcbe-4818-a7c5-de107988e30a-kube-api-access-8v64p" pod "f47ea4aa-bcbe-4818-a7c5-de107988e30a" (UID: "f47ea4aa-bcbe-4818-a7c5-de107988e30a"). InnerVolumeSpecName "kube-api-access-8v64p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:44.135302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.135262 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4219ddc1-da11-43be-abf9-d0de6302a37b-kube-api-access-s9tt9" pod "4219ddc1-da11-43be-abf9-d0de6302a37b" (UID: "4219ddc1-da11-43be-abf9-d0de6302a37b"). InnerVolumeSpecName "kube-api-access-s9tt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:44.139827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.139803 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-util" pod "4219ddc1-da11-43be-abf9-d0de6302a37b" (UID: "4219ddc1-da11-43be-abf9-d0de6302a37b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:44.140834 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.140775 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-util" pod "f47ea4aa-bcbe-4818-a7c5-de107988e30a" (UID: "f47ea4aa-bcbe-4818-a7c5-de107988e30a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:44.232718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.232696 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:44.232825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.232723 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9tt9\" (UniqueName: \"kubernetes.io/projected/4219ddc1-da11-43be-abf9-d0de6302a37b-kube-api-access-s9tt9\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:44.232825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.232733 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:44.232825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.232742 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4219ddc1-da11-43be-abf9-d0de6302a37b-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:44.232825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.232753 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v64p\" (UniqueName: \"kubernetes.io/projected/f47ea4aa-bcbe-4818-a7c5-de107988e30a-kube-api-access-8v64p\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:44.232825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.232764 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47ea4aa-bcbe-4818-a7c5-de107988e30a-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:44.274164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.272252 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" podStartSLOduration=3.945047233 podStartE2EDuration="52.272237535s" totalImagesPullingTime="48.327190302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:44.163103628 +0000 UTC m=+251.198398434" watchObservedRunningTime="2026-10-31 16:06:44.272237535 +0000 UTC m=+251.307532338" Oct 31 16:06:44.847029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.846947 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" event={"ID":"4219ddc1-da11-43be-abf9-d0de6302a37b","Type":"ContainerDied","Data":"4e78e9e4395ed1431be5db514e1d6544809e26c13a1ab6d1b365096cefa8b146"} Oct 31 16:06:44.847029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.847007 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e78e9e4395ed1431be5db514e1d6544809e26c13a1ab6d1b365096cefa8b146" Oct 31 16:06:44.847550 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.847080 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716btqm" Oct 31 16:06:44.854457 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.854382 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:06:44.855356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.855328 4880 generic.go:356] "Generic (PLEG): container finished" podID="874393ca-7c09-42ce-8149-d778433b0ea6" containerID="25f0460d499b38eedad9ef532484a1c12b418f152885536dd1f701c93cd846bd" exitCode=1 Oct 31 16:06:44.855480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.855431 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" event={"ID":"874393ca-7c09-42ce-8149-d778433b0ea6","Type":"ContainerDied","Data":"25f0460d499b38eedad9ef532484a1c12b418f152885536dd1f701c93cd846bd"} Oct 31 16:06:44.859758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.859458 4880 generic.go:356] "Generic (PLEG): container finished" podID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerID="3d558e124203cb97fc0c82d21a1a31e0a393bed6c46f73680ae498ac8f8bfc41" exitCode=0 Oct 31 16:06:44.859758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.859592 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" event={"ID":"66b944ef-1b01-4c39-965f-62c61f3ac460","Type":"ContainerDied","Data":"3d558e124203cb97fc0c82d21a1a31e0a393bed6c46f73680ae498ac8f8bfc41"} Oct 31 16:06:44.863236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.863209 4880 generic.go:356] "Generic (PLEG): container finished" podID="700ad351-7585-433f-9931-8ba86340af4d" containerID="91c65a93a94add8f10d956b03f73a89f573e692567e361419603c34942d5c82b" exitCode=0 Oct 31 16:06:44.863324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.863291 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" event={"ID":"700ad351-7585-433f-9931-8ba86340af4d","Type":"ContainerDied","Data":"91c65a93a94add8f10d956b03f73a89f573e692567e361419603c34942d5c82b"} Oct 31 16:06:44.866920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.866892 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" event={"ID":"f47ea4aa-bcbe-4818-a7c5-de107988e30a","Type":"ContainerDied","Data":"31434971b695fc72bfb136bd72e4cecf7f7d91c87bfc27076341ef06e6f349aa"} Oct 31 16:06:44.867074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.867056 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31434971b695fc72bfb136bd72e4cecf7f7d91c87bfc27076341ef06e6f349aa" Oct 31 16:06:44.867316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.867300 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71d25z2" Oct 31 16:06:44.874757 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.874736 4880 generic.go:356] "Generic (PLEG): container finished" podID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerID="e9cf735949dba28db6624dac0634d015a4b3956ba49afa1a55931ae9d6380e2f" exitCode=0 Oct 31 16:06:44.875036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:44.875014 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" event={"ID":"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082","Type":"ContainerDied","Data":"e9cf735949dba28db6624dac0634d015a4b3956ba49afa1a55931ae9d6380e2f"} Oct 31 16:06:45.040986 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.040956 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:06:45.112099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.112057 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:06:45.143713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.143687 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-bundle\") pod \"ff6b4072-258a-4a64-b900-141c82967bb6\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " Oct 31 16:06:45.143825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.143734 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-util\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-util\") pod \"ff6b4072-258a-4a64-b900-141c82967bb6\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " Oct 31 16:06:45.143825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.143796 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6b4072-258a-4a64-b900-141c82967bb6-kube-api-access-lsbdf\" (UniqueName: \"kubernetes.io/projected/ff6b4072-258a-4a64-b900-141c82967bb6-kube-api-access-lsbdf\") pod \"ff6b4072-258a-4a64-b900-141c82967bb6\" (UID: \"ff6b4072-258a-4a64-b900-141c82967bb6\") " Oct 31 16:06:45.144941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.144891 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-bundle" pod "ff6b4072-258a-4a64-b900-141c82967bb6" (UID: "ff6b4072-258a-4a64-b900-141c82967bb6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:45.152511 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.152478 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-util" pod "ff6b4072-258a-4a64-b900-141c82967bb6" (UID: "ff6b4072-258a-4a64-b900-141c82967bb6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:45.156361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.156310 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6b4072-258a-4a64-b900-141c82967bb6-kube-api-access-lsbdf" pod "ff6b4072-258a-4a64-b900-141c82967bb6" (UID: "ff6b4072-258a-4a64-b900-141c82967bb6"). InnerVolumeSpecName "kube-api-access-lsbdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:45.245059 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.245017 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-bundle\") pod \"b477a592-8028-4f37-b90f-641df89bad23\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " Oct 31 16:06:45.245059 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.245063 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-util\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-util\") pod \"b477a592-8028-4f37-b90f-641df89bad23\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " Oct 31 16:06:45.245312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.245112 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b477a592-8028-4f37-b90f-641df89bad23-kube-api-access-trf75\" (UniqueName: \"kubernetes.io/projected/b477a592-8028-4f37-b90f-641df89bad23-kube-api-access-trf75\") pod \"b477a592-8028-4f37-b90f-641df89bad23\" (UID: \"b477a592-8028-4f37-b90f-641df89bad23\") " Oct 31 16:06:45.245462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.245439 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:45.245462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.245460 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff6b4072-258a-4a64-b900-141c82967bb6-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:45.245616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.245470 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsbdf\" (UniqueName: \"kubernetes.io/projected/ff6b4072-258a-4a64-b900-141c82967bb6-kube-api-access-lsbdf\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:45.248600 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.248572 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-bundle" pod "b477a592-8028-4f37-b90f-641df89bad23" (UID: "b477a592-8028-4f37-b90f-641df89bad23"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:45.248734 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.248634 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b477a592-8028-4f37-b90f-641df89bad23-kube-api-access-trf75" pod "b477a592-8028-4f37-b90f-641df89bad23" (UID: "b477a592-8028-4f37-b90f-641df89bad23"). InnerVolumeSpecName "kube-api-access-trf75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:45.251991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.251967 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-util" pod "b477a592-8028-4f37-b90f-641df89bad23" (UID: "b477a592-8028-4f37-b90f-641df89bad23"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:45.346046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.346015 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:45.346046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.346041 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b477a592-8028-4f37-b90f-641df89bad23-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:45.346181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.346055 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trf75\" (UniqueName: \"kubernetes.io/projected/b477a592-8028-4f37-b90f-641df89bad23-kube-api-access-trf75\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:45.883360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.883338 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:06:45.884223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.884190 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" event={"ID":"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6","Type":"ContainerStarted","Data":"95f873952c4b730e90d8c38e2f1698ecbac7b2cd0de65bbb8b0c1d825fdfa0ef"} Oct 31 16:06:45.886406 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.886384 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" Oct 31 16:06:45.886510 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.886413 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cx56vh" event={"ID":"b477a592-8028-4f37-b90f-641df89bad23","Type":"ContainerDied","Data":"17659408e9bbc27a643dabc16dd0b320fe3bceae52933b02c424bab84b8275d9"} Oct 31 16:06:45.886510 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.886467 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17659408e9bbc27a643dabc16dd0b320fe3bceae52933b02c424bab84b8275d9" Oct 31 16:06:45.888898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.888870 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" event={"ID":"9938c0be-e9db-4358-8118-725a458b7a18","Type":"ContainerStarted","Data":"93f0364fccfe56d220eaae95db0ef769fc29a93225dcd5d0f654ab0d3c51c8a6"} Oct 31 16:06:45.891239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.891215 4880 generic.go:356] "Generic (PLEG): container finished" podID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerID="2fd8323becddfbbd5a60ebc06709a9068c10812d63430403f6c05b357746d9f6" exitCode=0 Oct 31 16:06:45.891351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.891331 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" event={"ID":"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5","Type":"ContainerDied","Data":"2fd8323becddfbbd5a60ebc06709a9068c10812d63430403f6c05b357746d9f6"} Oct 31 16:06:45.893575 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.893557 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" Oct 31 16:06:45.894462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.894225 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d764mrs" event={"ID":"ff6b4072-258a-4a64-b900-141c82967bb6","Type":"ContainerDied","Data":"56f75b9bb47f4e822ebfa7554ab0d4efb7bdf5027165a45290a0dd125ef2e482"} Oct 31 16:06:45.894462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:45.894273 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56f75b9bb47f4e822ebfa7554ab0d4efb7bdf5027165a45290a0dd125ef2e482" Oct 31 16:06:46.083674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.083648 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:06:46.101172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.101108 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:06:46.103208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.103190 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:06:46.110035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.110015 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:06:46.114799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.114780 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:06:46.157802 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.157778 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-util\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-util\") pod \"874393ca-7c09-42ce-8149-d778433b0ea6\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " Oct 31 16:06:46.157924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.157831 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-util\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-util\") pod \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " Oct 31 16:06:46.157924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.157885 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-bundle\") pod \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " Oct 31 16:06:46.157924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.157920 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-bundle\") pod \"874393ca-7c09-42ce-8149-d778433b0ea6\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " Oct 31 16:06:46.158042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.157951 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874393ca-7c09-42ce-8149-d778433b0ea6-kube-api-access-85znq\" (UniqueName: \"kubernetes.io/projected/874393ca-7c09-42ce-8149-d778433b0ea6-kube-api-access-85znq\") pod \"874393ca-7c09-42ce-8149-d778433b0ea6\" (UID: \"874393ca-7c09-42ce-8149-d778433b0ea6\") " Oct 31 16:06:46.158042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.158008 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-kube-api-access-8wwqr\" (UniqueName: \"kubernetes.io/projected/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-kube-api-access-8wwqr\") pod \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\" (UID: \"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082\") " Oct 31 16:06:46.158451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.158404 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-bundle" pod "04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" (UID: "04b792f2-6bb0-4da0-a3d8-84e7cf6e7082"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.158451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.158425 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-bundle" pod "874393ca-7c09-42ce-8149-d778433b0ea6" (UID: "874393ca-7c09-42ce-8149-d778433b0ea6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.160257 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.160225 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-kube-api-access-8wwqr" pod "04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" (UID: "04b792f2-6bb0-4da0-a3d8-84e7cf6e7082"). InnerVolumeSpecName "kube-api-access-8wwqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:46.160433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.160410 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874393ca-7c09-42ce-8149-d778433b0ea6-kube-api-access-85znq" pod "874393ca-7c09-42ce-8149-d778433b0ea6" (UID: "874393ca-7c09-42ce-8149-d778433b0ea6"). InnerVolumeSpecName "kube-api-access-85znq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:46.163197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.163178 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-util" pod "04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" (UID: "04b792f2-6bb0-4da0-a3d8-84e7cf6e7082"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.163673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.163653 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-util" pod "874393ca-7c09-42ce-8149-d778433b0ea6" (UID: "874393ca-7c09-42ce-8149-d778433b0ea6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.258809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.258789 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-bundle\") pod \"700ad351-7585-433f-9931-8ba86340af4d\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " Oct 31 16:06:46.258902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.258819 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700ad351-7585-433f-9931-8ba86340af4d-kube-api-access-bq6jj\" (UniqueName: \"kubernetes.io/projected/700ad351-7585-433f-9931-8ba86340af4d-kube-api-access-bq6jj\") pod \"700ad351-7585-433f-9931-8ba86340af4d\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " Oct 31 16:06:46.258902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.258860 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-util\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-util\") pod \"700ad351-7585-433f-9931-8ba86340af4d\" (UID: \"700ad351-7585-433f-9931-8ba86340af4d\") " Oct 31 16:06:46.258902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.258884 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-util\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-util\") pod \"66b944ef-1b01-4c39-965f-62c61f3ac460\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " Oct 31 16:06:46.259048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.258928 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b944ef-1b01-4c39-965f-62c61f3ac460-kube-api-access-857j7\" (UniqueName: \"kubernetes.io/projected/66b944ef-1b01-4c39-965f-62c61f3ac460-kube-api-access-857j7\") pod \"66b944ef-1b01-4c39-965f-62c61f3ac460\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " Oct 31 16:06:46.259102 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259057 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-bundle\") pod \"66b944ef-1b01-4c39-965f-62c61f3ac460\" (UID: \"66b944ef-1b01-4c39-965f-62c61f3ac460\") " Oct 31 16:06:46.259360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259339 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.259360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259361 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.259524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259373 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.259524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259385 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/874393ca-7c09-42ce-8149-d778433b0ea6-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.259524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259397 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85znq\" (UniqueName: \"kubernetes.io/projected/874393ca-7c09-42ce-8149-d778433b0ea6-kube-api-access-85znq\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.259524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259410 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wwqr\" (UniqueName: \"kubernetes.io/projected/04b792f2-6bb0-4da0-a3d8-84e7cf6e7082-kube-api-access-8wwqr\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.259524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259412 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-bundle" pod "700ad351-7585-433f-9931-8ba86340af4d" (UID: "700ad351-7585-433f-9931-8ba86340af4d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.259675 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.259594 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-bundle" pod "66b944ef-1b01-4c39-965f-62c61f3ac460" (UID: "66b944ef-1b01-4c39-965f-62c61f3ac460"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.261378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.261358 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700ad351-7585-433f-9931-8ba86340af4d-kube-api-access-bq6jj" pod "700ad351-7585-433f-9931-8ba86340af4d" (UID: "700ad351-7585-433f-9931-8ba86340af4d"). InnerVolumeSpecName "kube-api-access-bq6jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:46.261465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.261390 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b944ef-1b01-4c39-965f-62c61f3ac460-kube-api-access-857j7" pod "66b944ef-1b01-4c39-965f-62c61f3ac460" (UID: "66b944ef-1b01-4c39-965f-62c61f3ac460"). InnerVolumeSpecName "kube-api-access-857j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:46.264431 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.264404 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-util" pod "700ad351-7585-433f-9931-8ba86340af4d" (UID: "700ad351-7585-433f-9931-8ba86340af4d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.264681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.264663 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-util" pod "66b944ef-1b01-4c39-965f-62c61f3ac460" (UID: "66b944ef-1b01-4c39-965f-62c61f3ac460"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:46.308986 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.308943 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" podStartSLOduration=3.158727111 podStartE2EDuration="47.308930595s" totalImagesPullingTime="44.150203484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:46.308203126 +0000 UTC m=+253.343497929" watchObservedRunningTime="2026-10-31 16:06:46.308930595 +0000 UTC m=+253.344225400" Oct 31 16:06:46.359752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.359730 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.359752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.359754 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.359909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.359768 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-857j7\" (UniqueName: \"kubernetes.io/projected/66b944ef-1b01-4c39-965f-62c61f3ac460-kube-api-access-857j7\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.359909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.359781 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b944ef-1b01-4c39-965f-62c61f3ac460-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.359909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.359793 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/700ad351-7585-433f-9931-8ba86340af4d-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.359909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.359807 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6jj\" (UniqueName: \"kubernetes.io/projected/700ad351-7585-433f-9931-8ba86340af4d-kube-api-access-bq6jj\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:46.900381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.900304 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" Oct 31 16:06:46.900795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.900302 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67p7mb" event={"ID":"700ad351-7585-433f-9931-8ba86340af4d","Type":"ContainerDied","Data":"b8f928ceaf602adf83947ad3768c9207db1e25cc524bec37df0f3ced62e64c02"} Oct 31 16:06:46.900795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.900425 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f928ceaf602adf83947ad3768c9207db1e25cc524bec37df0f3ced62e64c02" Oct 31 16:06:46.901941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.901921 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:06:46.902564 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.902542 4880 generic.go:356] "Generic (PLEG): container finished" podID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerID="95f873952c4b730e90d8c38e2f1698ecbac7b2cd0de65bbb8b0c1d825fdfa0ef" exitCode=1 Oct 31 16:06:46.902655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.902600 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" event={"ID":"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6","Type":"ContainerDied","Data":"95f873952c4b730e90d8c38e2f1698ecbac7b2cd0de65bbb8b0c1d825fdfa0ef"} Oct 31 16:06:46.904774 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.904750 4880 generic.go:356] "Generic (PLEG): container finished" podID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerID="d0fdbcecf7713f1af4bd24186f0823dd155cbdc376ba4dd0349bc507f6ec3372" exitCode=0 Oct 31 16:06:46.904871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.904776 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" event={"ID":"f6494132-013c-40b9-b7c1-c3c852cee65c","Type":"ContainerDied","Data":"d0fdbcecf7713f1af4bd24186f0823dd155cbdc376ba4dd0349bc507f6ec3372"} Oct 31 16:06:46.906936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.906917 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" Oct 31 16:06:46.907064 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.906917 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mvg65" event={"ID":"04b792f2-6bb0-4da0-a3d8-84e7cf6e7082","Type":"ContainerDied","Data":"e2de99650658e74767d2ea5243aefb9a2d81949b44946edc3f8014ac79263e2c"} Oct 31 16:06:46.907195 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.907074 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2de99650658e74767d2ea5243aefb9a2d81949b44946edc3f8014ac79263e2c" Oct 31 16:06:46.908412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.908395 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:06:46.913182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.909437 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" Oct 31 16:06:46.913182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.909463 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs" event={"ID":"874393ca-7c09-42ce-8149-d778433b0ea6","Type":"ContainerDied","Data":"b05f680d8083bb17ecee164357006a89275c3f7ff0001c04bd8ebe6a5e743302"} Oct 31 16:06:46.913182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.909513 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b05f680d8083bb17ecee164357006a89275c3f7ff0001c04bd8ebe6a5e743302" Oct 31 16:06:46.913182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.911805 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" Oct 31 16:06:46.913182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.911815 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6s8vg" event={"ID":"66b944ef-1b01-4c39-965f-62c61f3ac460","Type":"ContainerDied","Data":"fba2ff03600b8223bfcfda04621d96435d41bb8ade15eb918ebca7ac83497922"} Oct 31 16:06:46.913182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.911849 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba2ff03600b8223bfcfda04621d96435d41bb8ade15eb918ebca7ac83497922" Oct 31 16:06:46.917051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.916910 4880 generic.go:356] "Generic (PLEG): container finished" podID="9938c0be-e9db-4358-8118-725a458b7a18" containerID="93f0364fccfe56d220eaae95db0ef769fc29a93225dcd5d0f654ab0d3c51c8a6" exitCode=0 Oct 31 16:06:46.917051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:46.917031 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" event={"ID":"9938c0be-e9db-4358-8118-725a458b7a18","Type":"ContainerDied","Data":"93f0364fccfe56d220eaae95db0ef769fc29a93225dcd5d0f654ab0d3c51c8a6"} Oct 31 16:06:47.029738 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.029716 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:06:47.031693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.031670 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:06:47.070458 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.070434 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:06:47.167764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.167684 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-kube-api-access-5mxlm\" (UniqueName: \"kubernetes.io/projected/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-kube-api-access-5mxlm\") pod \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " Oct 31 16:06:47.167764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.167729 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-kube-api-access-kgkdc\" (UniqueName: \"kubernetes.io/projected/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-kube-api-access-kgkdc\") pod \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " Oct 31 16:06:47.167935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.167813 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-bundle\") pod \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " Oct 31 16:06:47.167935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.167855 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-bundle\") pod \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " Oct 31 16:06:47.167935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.167896 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-util\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-util\") pod \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\" (UID: \"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5\") " Oct 31 16:06:47.168042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.167941 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-util\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-util\") pod \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\" (UID: \"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6\") " Oct 31 16:06:47.168588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.168553 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-bundle" pod "551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" (UID: "551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:47.168731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.168676 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-bundle" pod "1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" (UID: "1cd2c90f-11c8-46ce-bc99-7bcc746f69f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:47.169946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.169925 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-kube-api-access-5mxlm" pod "1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" (UID: "1cd2c90f-11c8-46ce-bc99-7bcc746f69f6"). InnerVolumeSpecName "kube-api-access-5mxlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:47.170215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.170197 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-kube-api-access-kgkdc" pod "551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" (UID: "551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5"). InnerVolumeSpecName "kube-api-access-kgkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:47.174193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.174170 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-util" pod "1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" (UID: "1cd2c90f-11c8-46ce-bc99-7bcc746f69f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:47.174458 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.174438 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-util" pod "551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" (UID: "551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:47.268877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.268844 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mxlm\" (UniqueName: \"kubernetes.io/projected/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-kube-api-access-5mxlm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:47.268877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.268872 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkdc\" (UniqueName: \"kubernetes.io/projected/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-kube-api-access-kgkdc\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:47.268877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.268883 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:47.269087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.268892 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:47.269087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.268900 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:47.269087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.268907 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd2c90f-11c8-46ce-bc99-7bcc746f69f6-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:47.923645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.923616 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:06:47.924462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.924439 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" Oct 31 16:06:47.924600 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.924459 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm" event={"ID":"1cd2c90f-11c8-46ce-bc99-7bcc746f69f6","Type":"ContainerDied","Data":"c4956fec8f57471c895463fc0df457a2e7930078dcf3b58c4d79e1b410baa709"} Oct 31 16:06:47.924600 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.924512 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4956fec8f57471c895463fc0df457a2e7930078dcf3b58c4d79e1b410baa709" Oct 31 16:06:47.926498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.926479 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" Oct 31 16:06:47.926498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.926479 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr462w" event={"ID":"551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5","Type":"ContainerDied","Data":"9e9c584f9bf600fe1afedcc0690db450c5c8c353aa91c445d8ad3b525c44551d"} Oct 31 16:06:47.926630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:47.926521 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e9c584f9bf600fe1afedcc0690db450c5c8c353aa91c445d8ad3b525c44551d" Oct 31 16:06:48.053521 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.053499 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:06:48.076833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.076812 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:06:48.177482 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.177456 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-bundle\") pod \"9938c0be-e9db-4358-8118-725a458b7a18\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " Oct 31 16:06:48.177621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.177521 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6494132-013c-40b9-b7c1-c3c852cee65c-kube-api-access-bx4tj\" (UniqueName: \"kubernetes.io/projected/f6494132-013c-40b9-b7c1-c3c852cee65c-kube-api-access-bx4tj\") pod \"f6494132-013c-40b9-b7c1-c3c852cee65c\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " Oct 31 16:06:48.177621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.177571 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-bundle\") pod \"f6494132-013c-40b9-b7c1-c3c852cee65c\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " Oct 31 16:06:48.177694 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.177623 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-util\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-util\") pod \"f6494132-013c-40b9-b7c1-c3c852cee65c\" (UID: \"f6494132-013c-40b9-b7c1-c3c852cee65c\") " Oct 31 16:06:48.177694 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.177672 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9938c0be-e9db-4358-8118-725a458b7a18-kube-api-access-hhbgn\" (UniqueName: \"kubernetes.io/projected/9938c0be-e9db-4358-8118-725a458b7a18-kube-api-access-hhbgn\") pod \"9938c0be-e9db-4358-8118-725a458b7a18\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " Oct 31 16:06:48.177795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.177715 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-util\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-util\") pod \"9938c0be-e9db-4358-8118-725a458b7a18\" (UID: \"9938c0be-e9db-4358-8118-725a458b7a18\") " Oct 31 16:06:48.178039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.178007 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-bundle" pod "f6494132-013c-40b9-b7c1-c3c852cee65c" (UID: "f6494132-013c-40b9-b7c1-c3c852cee65c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:48.178577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.178552 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-bundle" pod "9938c0be-e9db-4358-8118-725a458b7a18" (UID: "9938c0be-e9db-4358-8118-725a458b7a18"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:48.179855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.179830 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9938c0be-e9db-4358-8118-725a458b7a18-kube-api-access-hhbgn" pod "9938c0be-e9db-4358-8118-725a458b7a18" (UID: "9938c0be-e9db-4358-8118-725a458b7a18"). InnerVolumeSpecName "kube-api-access-hhbgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:48.179981 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.179963 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6494132-013c-40b9-b7c1-c3c852cee65c-kube-api-access-bx4tj" pod "f6494132-013c-40b9-b7c1-c3c852cee65c" (UID: "f6494132-013c-40b9-b7c1-c3c852cee65c"). InnerVolumeSpecName "kube-api-access-bx4tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:48.184307 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.184285 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-util" pod "f6494132-013c-40b9-b7c1-c3c852cee65c" (UID: "f6494132-013c-40b9-b7c1-c3c852cee65c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:48.185288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.185268 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-util" pod "9938c0be-e9db-4358-8118-725a458b7a18" (UID: "9938c0be-e9db-4358-8118-725a458b7a18"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:48.278673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.278613 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:48.278673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.278637 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhbgn\" (UniqueName: \"kubernetes.io/projected/9938c0be-e9db-4358-8118-725a458b7a18-kube-api-access-hhbgn\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:48.278673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.278647 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:48.278673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.278656 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9938c0be-e9db-4358-8118-725a458b7a18-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:48.278673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.278665 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4tj\" (UniqueName: \"kubernetes.io/projected/f6494132-013c-40b9-b7c1-c3c852cee65c-kube-api-access-bx4tj\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:48.278673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.278672 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6494132-013c-40b9-b7c1-c3c852cee65c-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:48.933843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.933814 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" Oct 31 16:06:48.933843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.933828 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448h79v" event={"ID":"f6494132-013c-40b9-b7c1-c3c852cee65c","Type":"ContainerDied","Data":"2fc4f8e7f18f90311d7f7dd71413570702c9e2536f2aeedcfca5bc7e15b84ef5"} Oct 31 16:06:48.934282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.933874 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc4f8e7f18f90311d7f7dd71413570702c9e2536f2aeedcfca5bc7e15b84ef5" Oct 31 16:06:48.935753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.935727 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" event={"ID":"9938c0be-e9db-4358-8118-725a458b7a18","Type":"ContainerDied","Data":"60e7ddc9508791b0a80c5c886b3597d1df5bac73151e20bac708ec4a909b0989"} Oct 31 16:06:48.935867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.935762 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5kp6z" Oct 31 16:06:48.935867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:48.935770 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e7ddc9508791b0a80c5c886b3597d1df5bac73151e20bac708ec4a909b0989" Oct 31 16:06:50.131811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:50.131783 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:50.398783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:50.398746 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:50.804745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:50.804668 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:51.000092 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.000055 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.017767 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw"] Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018087 4880 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b4072-258a-4a64-b900-141c82967bb6" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018098 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b4072-258a-4a64-b900-141c82967bb6" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018110 4880 cpu_manager.go:414] "containerMap: removing container" podUID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018116 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018124 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018129 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018164 4880 cpu_manager.go:414] "containerMap: removing container" podUID="874393ca-7c09-42ce-8149-d778433b0ea6" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018172 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874393ca-7c09-42ce-8149-d778433b0ea6" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018181 4880 cpu_manager.go:414] "containerMap: removing container" podUID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018185 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018194 4880 cpu_manager.go:414] "containerMap: removing container" podUID="700ad351-7585-433f-9931-8ba86340af4d" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018198 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700ad351-7585-433f-9931-8ba86340af4d" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018204 4880 cpu_manager.go:414] "containerMap: removing container" podUID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018208 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018214 4880 cpu_manager.go:414] "containerMap: removing container" podUID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018217 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018224 4880 cpu_manager.go:414] "containerMap: removing container" podUID="b477a592-8028-4f37-b90f-641df89bad23" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018229 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b477a592-8028-4f37-b90f-641df89bad23" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018236 4880 cpu_manager.go:414] "containerMap: removing container" podUID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018240 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018246 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018250 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018257 4880 cpu_manager.go:414] "containerMap: removing container" podUID="9938c0be-e9db-4358-8118-725a458b7a18" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018261 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9938c0be-e9db-4358-8118-725a458b7a18" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018267 4880 cpu_manager.go:414] "containerMap: removing container" podUID="874393ca-7c09-42ce-8149-d778433b0ea6" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018271 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874393ca-7c09-42ce-8149-d778433b0ea6" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018277 4880 cpu_manager.go:414] "containerMap: removing container" podUID="b477a592-8028-4f37-b90f-641df89bad23" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018282 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b477a592-8028-4f37-b90f-641df89bad23" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018288 4880 cpu_manager.go:414] "containerMap: removing container" podUID="700ad351-7585-433f-9931-8ba86340af4d" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018292 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700ad351-7585-433f-9931-8ba86340af4d" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018297 4880 cpu_manager.go:414] "containerMap: removing container" podUID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018302 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018308 4880 cpu_manager.go:414] "containerMap: removing container" podUID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018312 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018318 4880 cpu_manager.go:414] "containerMap: removing container" podUID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018322 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018329 4880 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b4072-258a-4a64-b900-141c82967bb6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018333 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b4072-258a-4a64-b900-141c82967bb6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018354 4880 cpu_manager.go:414] "containerMap: removing container" podUID="ff6b4072-258a-4a64-b900-141c82967bb6" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018358 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6b4072-258a-4a64-b900-141c82967bb6" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018363 4880 cpu_manager.go:414] "containerMap: removing container" podUID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018368 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018374 4880 cpu_manager.go:414] "containerMap: removing container" podUID="9938c0be-e9db-4358-8118-725a458b7a18" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018378 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9938c0be-e9db-4358-8118-725a458b7a18" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018383 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018387 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018393 4880 cpu_manager.go:414] "containerMap: removing container" podUID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018397 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018402 4880 cpu_manager.go:414] "containerMap: removing container" podUID="700ad351-7585-433f-9931-8ba86340af4d" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018406 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700ad351-7585-433f-9931-8ba86340af4d" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018411 4880 cpu_manager.go:414] "containerMap: removing container" podUID="9938c0be-e9db-4358-8118-725a458b7a18" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018415 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9938c0be-e9db-4358-8118-725a458b7a18" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018421 4880 cpu_manager.go:414] "containerMap: removing container" podUID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018425 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018433 4880 cpu_manager.go:414] "containerMap: removing container" podUID="b477a592-8028-4f37-b90f-641df89bad23" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018438 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b477a592-8028-4f37-b90f-641df89bad23" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018445 4880 cpu_manager.go:414] "containerMap: removing container" podUID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018449 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018454 4880 cpu_manager.go:414] "containerMap: removing container" podUID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018460 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018465 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018469 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerName="pull" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018476 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018480 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018486 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018491 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018496 4880 cpu_manager.go:414] "containerMap: removing container" podUID="874393ca-7c09-42ce-8149-d778433b0ea6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018500 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874393ca-7c09-42ce-8149-d778433b0ea6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018506 4880 cpu_manager.go:414] "containerMap: removing container" podUID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018510 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018517 4880 cpu_manager.go:414] "containerMap: removing container" podUID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018521 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerName="util" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018570 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b792f2-6bb0-4da0-a3d8-84e7cf6e7082" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018579 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47ea4aa-bcbe-4818-a7c5-de107988e30a" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018585 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="874393ca-7c09-42ce-8149-d778433b0ea6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018590 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="551d37f5-a4f2-45ae-b26e-c6a35fdcc5b5" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018597 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b944ef-1b01-4c39-965f-62c61f3ac460" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018603 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="b477a592-8028-4f37-b90f-641df89bad23" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018609 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="700ad351-7585-433f-9931-8ba86340af4d" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018614 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd2c90f-11c8-46ce-bc99-7bcc746f69f6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018619 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="4219ddc1-da11-43be-abf9-d0de6302a37b" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018624 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6b4072-258a-4a64-b900-141c82967bb6" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018629 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="9938c0be-e9db-4358-8118-725a458b7a18" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.018634 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6494132-013c-40b9-b7c1-c3c852cee65c" containerName="extract" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.037099 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.087525 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.087980 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.088007 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zc6f5\"" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.088054 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.088068 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.088070 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.137288 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw"] Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.205684 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.205784 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b541f84-c65f-4bb6-88b9-ee262c6266bf-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.205811 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqfh6\" (UniqueName: \"kubernetes.io/projected/8b541f84-c65f-4bb6-88b9-ee262c6266bf-kube-api-access-bqfh6\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.205836 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.205936 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b541f84-c65f-4bb6-88b9-ee262c6266bf-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.205974 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.304483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.235533 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:51.308034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.307320 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b541f84-c65f-4bb6-88b9-ee262c6266bf-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.308034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.307355 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqfh6\" (UniqueName: \"kubernetes.io/projected/8b541f84-c65f-4bb6-88b9-ee262c6266bf-kube-api-access-bqfh6\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.308034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.307380 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.308034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.307416 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b541f84-c65f-4bb6-88b9-ee262c6266bf-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.308034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.307442 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.308034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.307486 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.308034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.307785 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b541f84-c65f-4bb6-88b9-ee262c6266bf-manifests\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.309695 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.309670 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b541f84-c65f-4bb6-88b9-ee262c6266bf-tmp\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.309834 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.309803 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.310123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.310106 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-metrics-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.310287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.310268 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b541f84-c65f-4bb6-88b9-ee262c6266bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.360538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.360511 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqfh6\" (UniqueName: \"kubernetes.io/projected/8b541f84-c65f-4bb6-88b9-ee262c6266bf-kube-api-access-bqfh6\") pod \"opendatahub-operator-controller-manager-54f884c9ff-tqgjw\" (UID: \"8b541f84-c65f-4bb6-88b9-ee262c6266bf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.372339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.372319 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:51.389431 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.389392 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:51.565913 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.565837 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:51.908977 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:06:51.908950 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b541f84_c65f_4bb6_88b9_ee262c6266bf.slice/crio-c9fc2513c286a0afd50a137c354d084d9800baea1a8da6028b4c4562137f063a WatchSource:0}: Error finding container c9fc2513c286a0afd50a137c354d084d9800baea1a8da6028b4c4562137f063a: Status 404 returned error can't find the container with id c9fc2513c286a0afd50a137c354d084d9800baea1a8da6028b4c4562137f063a Oct 31 16:06:51.943784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.943758 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw"] Oct 31 16:06:51.951331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:51.951299 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" event={"ID":"8b541f84-c65f-4bb6-88b9-ee262c6266bf","Type":"ContainerStarted","Data":"c9fc2513c286a0afd50a137c354d084d9800baea1a8da6028b4c4562137f063a"} Oct 31 16:06:53.789714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:53.789674 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:53.991290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:53.991255 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:54.047887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.047812 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf"] Oct 31 16:06:54.050661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.050632 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.130729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.130700 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.130969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.130756 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.130969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.130918 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh62d\" (UniqueName: \"kubernetes.io/projected/8153c889-156c-4b02-8482-ac2d0b16e33e-kube-api-access-mh62d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.148517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.148490 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-fhlfh\"" Oct 31 16:06:54.148828 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.148802 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 16:06:54.148964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.148939 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 16:06:54.151282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.151261 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf"] Oct 31 16:06:54.172074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.172046 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:54.232299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.232267 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh62d\" (UniqueName: \"kubernetes.io/projected/8153c889-156c-4b02-8482-ac2d0b16e33e-kube-api-access-mh62d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.232446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.232328 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.232446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.232368 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.232784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.232760 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.232784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.232773 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.287530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.287494 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh62d\" (UniqueName: \"kubernetes.io/projected/8153c889-156c-4b02-8482-ac2d0b16e33e-kube-api-access-mh62d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.329695 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.329603 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:54.367209 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.367181 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:54.735068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.735036 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf"] Oct 31 16:06:54.738094 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:06:54.738069 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8153c889_156c_4b02_8482_ac2d0b16e33e.slice/crio-572c379b4d27410a2945ca19abab998373d72b0a5143a7e2341f76f28bba378d WatchSource:0}: Error finding container 572c379b4d27410a2945ca19abab998373d72b0a5143a7e2341f76f28bba378d: Status 404 returned error can't find the container with id 572c379b4d27410a2945ca19abab998373d72b0a5143a7e2341f76f28bba378d Oct 31 16:06:54.969011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.968976 4880 generic.go:356] "Generic (PLEG): container finished" podID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerID="6ebc150f89960c6c32217d0abfec2d1b3015eda72cfb80ad140edc2bced5723a" exitCode=0 Oct 31 16:06:54.969512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.969041 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" event={"ID":"8153c889-156c-4b02-8482-ac2d0b16e33e","Type":"ContainerDied","Data":"6ebc150f89960c6c32217d0abfec2d1b3015eda72cfb80ad140edc2bced5723a"} Oct 31 16:06:54.969512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:54.969112 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" event={"ID":"8153c889-156c-4b02-8482-ac2d0b16e33e","Type":"ContainerStarted","Data":"572c379b4d27410a2945ca19abab998373d72b0a5143a7e2341f76f28bba378d"} Oct 31 16:06:55.974882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:55.974851 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" event={"ID":"8b541f84-c65f-4bb6-88b9-ee262c6266bf","Type":"ContainerStarted","Data":"0246dababc21861a19bcee669c18a5455c9436dc3e4d6196fccac07794c44016"} Oct 31 16:06:55.976647 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:55.976621 4880 generic.go:356] "Generic (PLEG): container finished" podID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerID="f4b8b2dd8e4015ed0e1a257a774ca5c1ebe0009fb1b13d0b6197f73986259cbb" exitCode=0 Oct 31 16:06:55.976782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:55.976706 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" event={"ID":"8153c889-156c-4b02-8482-ac2d0b16e33e","Type":"ContainerDied","Data":"f4b8b2dd8e4015ed0e1a257a774ca5c1ebe0009fb1b13d0b6197f73986259cbb"} Oct 31 16:06:56.983206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:56.983168 4880 generic.go:356] "Generic (PLEG): container finished" podID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerID="0a6534222085a5482c500dcb6efd218e1cb45afcd7aa35026f21c6756d6ccf0d" exitCode=0 Oct 31 16:06:56.983621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:56.983251 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" event={"ID":"8153c889-156c-4b02-8482-ac2d0b16e33e","Type":"ContainerDied","Data":"0a6534222085a5482c500dcb6efd218e1cb45afcd7aa35026f21c6756d6ccf0d"} Oct 31 16:06:56.984752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:56.984730 4880 generic.go:356] "Generic (PLEG): container finished" podID="8b541f84-c65f-4bb6-88b9-ee262c6266bf" containerID="0246dababc21861a19bcee669c18a5455c9436dc3e4d6196fccac07794c44016" exitCode=0 Oct 31 16:06:56.984877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:56.984756 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" event={"ID":"8b541f84-c65f-4bb6-88b9-ee262c6266bf","Type":"ContainerDied","Data":"0246dababc21861a19bcee669c18a5455c9436dc3e4d6196fccac07794c44016"} Oct 31 16:06:58.111684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.111659 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:58.267706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.267623 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-bundle\") pod \"8153c889-156c-4b02-8482-ac2d0b16e33e\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " Oct 31 16:06:58.267858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.267729 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8153c889-156c-4b02-8482-ac2d0b16e33e-kube-api-access-mh62d\" (UniqueName: \"kubernetes.io/projected/8153c889-156c-4b02-8482-ac2d0b16e33e-kube-api-access-mh62d\") pod \"8153c889-156c-4b02-8482-ac2d0b16e33e\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " Oct 31 16:06:58.267858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.267755 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-util\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-util\") pod \"8153c889-156c-4b02-8482-ac2d0b16e33e\" (UID: \"8153c889-156c-4b02-8482-ac2d0b16e33e\") " Oct 31 16:06:58.268241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.268213 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-bundle" pod "8153c889-156c-4b02-8482-ac2d0b16e33e" (UID: "8153c889-156c-4b02-8482-ac2d0b16e33e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:58.269777 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.269757 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8153c889-156c-4b02-8482-ac2d0b16e33e-kube-api-access-mh62d" pod "8153c889-156c-4b02-8482-ac2d0b16e33e" (UID: "8153c889-156c-4b02-8482-ac2d0b16e33e"). InnerVolumeSpecName "kube-api-access-mh62d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:58.276830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.276748 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-util" pod "8153c889-156c-4b02-8482-ac2d0b16e33e" (UID: "8153c889-156c-4b02-8482-ac2d0b16e33e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:06:58.368997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.368956 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh62d\" (UniqueName: \"kubernetes.io/projected/8153c889-156c-4b02-8482-ac2d0b16e33e-kube-api-access-mh62d\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:58.368997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.368988 4880 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-util\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:58.368997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.368997 4880 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8153c889-156c-4b02-8482-ac2d0b16e33e-bundle\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:06:58.995969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.995933 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" event={"ID":"8b541f84-c65f-4bb6-88b9-ee262c6266bf","Type":"ContainerStarted","Data":"b50fc75ae5afd10279c5e3b5c80d7a9d4ab36e4b6cbd90dcc54d4b60fe872ef3"} Oct 31 16:06:58.996176 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.996102 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:06:58.997790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.997771 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" Oct 31 16:06:58.997902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.997811 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v2czf" event={"ID":"8153c889-156c-4b02-8482-ac2d0b16e33e","Type":"ContainerDied","Data":"572c379b4d27410a2945ca19abab998373d72b0a5143a7e2341f76f28bba378d"} Oct 31 16:06:58.997902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:58.997861 4880 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="572c379b4d27410a2945ca19abab998373d72b0a5143a7e2341f76f28bba378d" Oct 31 16:06:59.169801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:59.169758 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:59.255882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:59.255737 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" podStartSLOduration=4.060789748 podStartE2EDuration="9.255722607s" totalImagesPullingTime="4.771610215s" totalInitContainerRuntime="423.322644ms" isStatefulPod=false podCreationTimestamp="2026-10-31 16:06:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:06:59.208375552 +0000 UTC m=+266.243670356" watchObservedRunningTime="2026-10-31 16:06:59.255722607 +0000 UTC m=+266.291017422" Oct 31 16:06:59.333041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:59.333000 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:06:59.486842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:06:59.486808 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:02.327297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:02.327261 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:03.437975 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:03.437943 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:03.880227 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:07:03.880125 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:07:05.303381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303344 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt"] Oct 31 16:07:05.303764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303651 4880 cpu_manager.go:414] "containerMap: removing container" podUID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerName="util" Oct 31 16:07:05.303764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303663 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerName="util" Oct 31 16:07:05.303764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303687 4880 cpu_manager.go:414] "containerMap: removing container" podUID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerName="extract" Oct 31 16:07:05.303764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303692 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerName="extract" Oct 31 16:07:05.303764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303703 4880 cpu_manager.go:414] "containerMap: removing container" podUID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerName="pull" Oct 31 16:07:05.303764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303709 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerName="pull" Oct 31 16:07:05.303764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.303766 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="8153c889-156c-4b02-8482-ac2d0b16e33e" containerName="extract" Oct 31 16:07:05.309786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.309765 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:05.354871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.354839 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt"] Oct 31 16:07:05.398780 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.398741 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 16:07:05.399056 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.399035 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:07:05.399164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.399124 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xch5g\"" Oct 31 16:07:05.431066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.431024 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dt6z\" (UniqueName: \"kubernetes.io/projected/8618cf9d-7e5b-4bbc-9ab9-f85b328d2021-kube-api-access-7dt6z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt\" (UID: \"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:05.431306 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.431073 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8618cf9d-7e5b-4bbc-9ab9-f85b328d2021-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt\" (UID: \"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:05.531947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.531907 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dt6z\" (UniqueName: \"kubernetes.io/projected/8618cf9d-7e5b-4bbc-9ab9-f85b328d2021-kube-api-access-7dt6z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt\" (UID: \"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:05.531947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.531947 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8618cf9d-7e5b-4bbc-9ab9-f85b328d2021-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt\" (UID: \"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:05.532318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.532301 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8618cf9d-7e5b-4bbc-9ab9-f85b328d2021-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt\" (UID: \"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:05.584976 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.584904 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dt6z\" (UniqueName: \"kubernetes.io/projected/8618cf9d-7e5b-4bbc-9ab9-f85b328d2021-kube-api-access-7dt6z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt\" (UID: \"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:05.620738 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:05.620704 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" Oct 31 16:07:06.082542 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:06.082515 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt"] Oct 31 16:07:06.084852 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:06.084825 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8618cf9d_7e5b_4bbc_9ab9_f85b328d2021.slice/crio-8eec970f29b58e33a94400636a87f3c1f195ef08886fd31ec54f285d8ca040dd WatchSource:0}: Error finding container 8eec970f29b58e33a94400636a87f3c1f195ef08886fd31ec54f285d8ca040dd: Status 404 returned error can't find the container with id 8eec970f29b58e33a94400636a87f3c1f195ef08886fd31ec54f285d8ca040dd Oct 31 16:07:07.035883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:07.035845 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" event={"ID":"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021","Type":"ContainerStarted","Data":"8eec970f29b58e33a94400636a87f3c1f195ef08886fd31ec54f285d8ca040dd"} Oct 31 16:07:10.005491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:10.005462 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54f884c9ff-tqgjw" Oct 31 16:07:10.052569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:10.052533 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" event={"ID":"8618cf9d-7e5b-4bbc-9ab9-f85b328d2021","Type":"ContainerStarted","Data":"f8579485f956bc85b8249ab73ab7fb8769b4105539de26e5960b73886442411f"} Oct 31 16:07:10.327811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:10.327694 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xl4qt" podStartSLOduration=2.389151008 podStartE2EDuration="5.327675525s" totalImagesPullingTime="2.938524517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:10.327403864 +0000 UTC m=+277.362698681" watchObservedRunningTime="2026-10-31 16:07:10.327675525 +0000 UTC m=+277.362970330" Oct 31 16:07:11.161291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:11.161253 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:11.608248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:11.608163 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:11.770287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:11.770245 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:12.127893 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.127838 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:12.359409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.359379 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk"] Oct 31 16:07:12.362950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.362929 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:12.387183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.387154 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14405e4c-cada-487b-89a2-f770fcb8c907-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xrfgk\" (UID: \"14405e4c-cada-487b-89a2-f770fcb8c907\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:12.387310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.387190 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpv6c\" (UniqueName: \"kubernetes.io/projected/14405e4c-cada-487b-89a2-f770fcb8c907-kube-api-access-rpv6c\") pod \"cert-manager-webhook-5dbbbf75cc-xrfgk\" (UID: \"14405e4c-cada-487b-89a2-f770fcb8c907\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:12.417346 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.417316 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 16:07:12.463468 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.463436 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 16:07:12.465822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.465800 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk"] Oct 31 16:07:12.488521 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.488490 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14405e4c-cada-487b-89a2-f770fcb8c907-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xrfgk\" (UID: \"14405e4c-cada-487b-89a2-f770fcb8c907\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:12.488698 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.488545 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpv6c\" (UniqueName: \"kubernetes.io/projected/14405e4c-cada-487b-89a2-f770fcb8c907-kube-api-access-rpv6c\") pod \"cert-manager-webhook-5dbbbf75cc-xrfgk\" (UID: \"14405e4c-cada-487b-89a2-f770fcb8c907\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:12.514568 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.514540 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ht5dq\"" Oct 31 16:07:12.549097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.549050 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/14405e4c-cada-487b-89a2-f770fcb8c907-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-xrfgk\" (UID: \"14405e4c-cada-487b-89a2-f770fcb8c907\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:12.550562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.550541 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpv6c\" (UniqueName: \"kubernetes.io/projected/14405e4c-cada-487b-89a2-f770fcb8c907-kube-api-access-rpv6c\") pod \"cert-manager-webhook-5dbbbf75cc-xrfgk\" (UID: \"14405e4c-cada-487b-89a2-f770fcb8c907\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:12.682838 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:12.682756 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:13.052370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.052297 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk"] Oct 31 16:07:13.055761 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:13.055735 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14405e4c_cada_487b_89a2_f770fcb8c907.slice/crio-ebee21ed38d330c7a19b44f9de1a3b283156b453d745ff5d9a97b989530c08b3 WatchSource:0}: Error finding container ebee21ed38d330c7a19b44f9de1a3b283156b453d745ff5d9a97b989530c08b3: Status 404 returned error can't find the container with id ebee21ed38d330c7a19b44f9de1a3b283156b453d745ff5d9a97b989530c08b3 Oct 31 16:07:13.070239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.070213 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" event={"ID":"14405e4c-cada-487b-89a2-f770fcb8c907","Type":"ContainerStarted","Data":"ebee21ed38d330c7a19b44f9de1a3b283156b453d745ff5d9a97b989530c08b3"} Oct 31 16:07:13.144455 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.144420 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:13.369912 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.369822 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6"] Oct 31 16:07:13.374073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.374046 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:13.397939 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.397907 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjm4\" (UniqueName: \"kubernetes.io/projected/34363e1b-3eda-451d-9654-89234db994b9-kube-api-access-tpjm4\") pod \"cert-manager-cainjector-5c57bbd4fc-jqdq6\" (UID: \"34363e1b-3eda-451d-9654-89234db994b9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:13.398066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.397967 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34363e1b-3eda-451d-9654-89234db994b9-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jqdq6\" (UID: \"34363e1b-3eda-451d-9654-89234db994b9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:13.428630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.428602 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9pxj6\"" Oct 31 16:07:13.445643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.445614 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6"] Oct 31 16:07:13.492632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.492596 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:13.498626 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.498597 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjm4\" (UniqueName: \"kubernetes.io/projected/34363e1b-3eda-451d-9654-89234db994b9-kube-api-access-tpjm4\") pod \"cert-manager-cainjector-5c57bbd4fc-jqdq6\" (UID: \"34363e1b-3eda-451d-9654-89234db994b9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:13.498739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.498657 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34363e1b-3eda-451d-9654-89234db994b9-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jqdq6\" (UID: \"34363e1b-3eda-451d-9654-89234db994b9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:13.552673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.552646 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34363e1b-3eda-451d-9654-89234db994b9-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jqdq6\" (UID: \"34363e1b-3eda-451d-9654-89234db994b9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:13.552811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.552733 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjm4\" (UniqueName: \"kubernetes.io/projected/34363e1b-3eda-451d-9654-89234db994b9-kube-api-access-tpjm4\") pod \"cert-manager-cainjector-5c57bbd4fc-jqdq6\" (UID: \"34363e1b-3eda-451d-9654-89234db994b9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:13.690491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:13.690455 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" Oct 31 16:07:14.055028 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:14.055003 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6"] Oct 31 16:07:14.057039 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:14.057008 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34363e1b_3eda_451d_9654_89234db994b9.slice/crio-86d7227f3e93a17a57f9fb43f435e883dafacee88fa27c68b8fab9382af4a142 WatchSource:0}: Error finding container 86d7227f3e93a17a57f9fb43f435e883dafacee88fa27c68b8fab9382af4a142: Status 404 returned error can't find the container with id 86d7227f3e93a17a57f9fb43f435e883dafacee88fa27c68b8fab9382af4a142 Oct 31 16:07:14.076315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:14.076284 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" event={"ID":"34363e1b-3eda-451d-9654-89234db994b9","Type":"ContainerStarted","Data":"86d7227f3e93a17a57f9fb43f435e883dafacee88fa27c68b8fab9382af4a142"} Oct 31 16:07:14.298872 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:14.298823 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:17.027837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.027800 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh"] Oct 31 16:07:17.031417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.031395 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.078673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.078641 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh"] Oct 31 16:07:17.122203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.122154 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 16:07:17.122203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.122163 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 16:07:17.122203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.122168 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qkljv\"" Oct 31 16:07:17.132970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.132945 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6p82\" (UniqueName: \"kubernetes.io/projected/f00d7cbc-2929-4309-acc5-ffc348c41a85-kube-api-access-q6p82\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6wxxh\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.133084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.132992 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00d7cbc-2929-4309-acc5-ffc348c41a85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6wxxh\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.211070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.211047 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:17.234009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.233975 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6p82\" (UniqueName: \"kubernetes.io/projected/f00d7cbc-2929-4309-acc5-ffc348c41a85-kube-api-access-q6p82\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6wxxh\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.234192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.234024 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00d7cbc-2929-4309-acc5-ffc348c41a85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6wxxh\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.234465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.234444 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00d7cbc-2929-4309-acc5-ffc348c41a85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6wxxh\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.290474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.290395 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6p82\" (UniqueName: \"kubernetes.io/projected/f00d7cbc-2929-4309-acc5-ffc348c41a85-kube-api-access-q6p82\") pod \"kuadrant-operator-controller-manager-cf8469d7b-6wxxh\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.345595 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.345564 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:17.375061 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.375021 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:17.730257 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.730219 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:17.767282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:17.767255 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh"] Oct 31 16:07:17.769346 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:17.769315 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00d7cbc_2929_4309_acc5_ffc348c41a85.slice/crio-5d7345f3476e09379cd88284b862706e3ffa9b880c8a8dab7710f869345483ad WatchSource:0}: Error finding container 5d7345f3476e09379cd88284b862706e3ffa9b880c8a8dab7710f869345483ad: Status 404 returned error can't find the container with id 5d7345f3476e09379cd88284b862706e3ffa9b880c8a8dab7710f869345483ad Oct 31 16:07:18.107947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:18.107658 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" event={"ID":"14405e4c-cada-487b-89a2-f770fcb8c907","Type":"ContainerStarted","Data":"1946214376e8e5c08db4710756b441e0581821f1195103f5f3531cb45ab88949"} Oct 31 16:07:18.107947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:18.107770 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:18.109520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:18.109493 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" event={"ID":"34363e1b-3eda-451d-9654-89234db994b9","Type":"ContainerStarted","Data":"1392753e96d08b3d24e7b06efc1ce0f3dc01ee45d0d00601e778c1924090f54d"} Oct 31 16:07:18.110764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:18.110739 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" event={"ID":"f00d7cbc-2929-4309-acc5-ffc348c41a85","Type":"ContainerStarted","Data":"5d7345f3476e09379cd88284b862706e3ffa9b880c8a8dab7710f869345483ad"} Oct 31 16:07:18.215934 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:18.215877 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" podStartSLOduration=2.210843567 podStartE2EDuration="6.215862322s" totalImagesPullingTime="4.005018755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:18.21452963 +0000 UTC m=+285.249824433" watchObservedRunningTime="2026-10-31 16:07:18.215862322 +0000 UTC m=+285.251157128" Oct 31 16:07:18.320509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:18.320458 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jqdq6" podStartSLOduration=2.220279876 podStartE2EDuration="5.3204443s" totalImagesPullingTime="3.100164424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:18.31940012 +0000 UTC m=+285.354694923" watchObservedRunningTime="2026-10-31 16:07:18.3204443 +0000 UTC m=+285.355739104" Oct 31 16:07:19.179437 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.179393 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt"] Oct 31 16:07:19.184075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.184054 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" Oct 31 16:07:19.241755 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.241711 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:19.250754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.250731 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45g4\" (UniqueName: \"kubernetes.io/projected/ecf4590e-572d-4c90-ad77-e130e762b449-kube-api-access-c45g4\") pod \"limitador-operator-controller-manager-5778bb4449-f5vdt\" (UID: \"ecf4590e-572d-4c90-ad77-e130e762b449\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" Oct 31 16:07:19.282113 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.282081 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jn9w6\"" Oct 31 16:07:19.284011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.283986 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt"] Oct 31 16:07:19.351729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.351702 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45g4\" (UniqueName: \"kubernetes.io/projected/ecf4590e-572d-4c90-ad77-e130e762b449-kube-api-access-c45g4\") pod \"limitador-operator-controller-manager-5778bb4449-f5vdt\" (UID: \"ecf4590e-572d-4c90-ad77-e130e762b449\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" Oct 31 16:07:19.387719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.387686 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:19.404478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.404456 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45g4\" (UniqueName: \"kubernetes.io/projected/ecf4590e-572d-4c90-ad77-e130e762b449-kube-api-access-c45g4\") pod \"limitador-operator-controller-manager-5778bb4449-f5vdt\" (UID: \"ecf4590e-572d-4c90-ad77-e130e762b449\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" Oct 31 16:07:19.497102 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.497038 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" Oct 31 16:07:19.748677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.748597 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:19.859878 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.859848 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt"] Oct 31 16:07:19.892681 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:19.892644 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf4590e_572d_4c90_ad77_e130e762b449.slice/crio-87bdad69c0449a27009dad7546c4ac2c51996ff94b951b9dc1010026710beaaf WatchSource:0}: Error finding container 87bdad69c0449a27009dad7546c4ac2c51996ff94b951b9dc1010026710beaaf: Status 404 returned error can't find the container with id 87bdad69c0449a27009dad7546c4ac2c51996ff94b951b9dc1010026710beaaf Oct 31 16:07:19.964927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:19.964897 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:20.122033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:20.121963 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" event={"ID":"ecf4590e-572d-4c90-ad77-e130e762b449","Type":"ContainerStarted","Data":"87bdad69c0449a27009dad7546c4ac2c51996ff94b951b9dc1010026710beaaf"} Oct 31 16:07:21.004057 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:21.004021 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:21.724159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:21.724094 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:22.146033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:22.145947 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:23.019879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:23.019825 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:23.189710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:23.189675 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:23.513297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:23.513266 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:24.117734 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.117662 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" Oct 31 16:07:24.148453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.148413 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" event={"ID":"f00d7cbc-2929-4309-acc5-ffc348c41a85","Type":"ContainerStarted","Data":"4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d"} Oct 31 16:07:24.148607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.148483 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:24.150123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.150092 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" event={"ID":"ecf4590e-572d-4c90-ad77-e130e762b449","Type":"ContainerStarted","Data":"911040ab7472acd35c254280538e33cf57f2f7bc2fd9a3cf296e5c389e52e860"} Oct 31 16:07:24.150250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.150195 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" Oct 31 16:07:24.339885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.339824 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" podStartSLOduration=1.820009821 podStartE2EDuration="5.339808452s" totalImagesPullingTime="3.519798631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:24.336344785 +0000 UTC m=+291.371639589" watchObservedRunningTime="2026-10-31 16:07:24.339808452 +0000 UTC m=+291.375103267" Oct 31 16:07:24.446166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.446097 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" podStartSLOduration=2.464694201 podStartE2EDuration="8.446082545s" totalImagesPullingTime="5.981388344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:24.443655251 +0000 UTC m=+291.478950054" watchObservedRunningTime="2026-10-31 16:07:24.446082545 +0000 UTC m=+291.481377350" Oct 31 16:07:24.500432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.500399 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh"] Oct 31 16:07:24.504109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.504084 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:07:24.556098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.556065 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-54wsx\"" Oct 31 16:07:24.559748 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.559721 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh"] Oct 31 16:07:24.595547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.595518 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwnj9\" (UniqueName: \"kubernetes.io/projected/cb9fb5ff-4889-4cf7-aa3a-37a80b636932-kube-api-access-pwnj9\") pod \"authorino-operator-7b7bb4bc77-8qmlh\" (UID: \"cb9fb5ff-4889-4cf7-aa3a-37a80b636932\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:07:24.696440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.696358 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwnj9\" (UniqueName: \"kubernetes.io/projected/cb9fb5ff-4889-4cf7-aa3a-37a80b636932-kube-api-access-pwnj9\") pod \"authorino-operator-7b7bb4bc77-8qmlh\" (UID: \"cb9fb5ff-4889-4cf7-aa3a-37a80b636932\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:07:24.751407 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.751372 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwnj9\" (UniqueName: \"kubernetes.io/projected/cb9fb5ff-4889-4cf7-aa3a-37a80b636932-kube-api-access-pwnj9\") pod \"authorino-operator-7b7bb4bc77-8qmlh\" (UID: \"cb9fb5ff-4889-4cf7-aa3a-37a80b636932\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:07:24.757755 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.757723 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:24.816904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:24.816873 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:07:25.169973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.169938 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:25.201781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.201755 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh"] Oct 31 16:07:25.203918 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:25.203891 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9fb5ff_4889_4cf7_aa3a_37a80b636932.slice/crio-b31f8111d1d2ceab3428f35df520623d348ffbbbd0060660c55fe3abbb68698d WatchSource:0}: Error finding container b31f8111d1d2ceab3428f35df520623d348ffbbbd0060660c55fe3abbb68698d: Status 404 returned error can't find the container with id b31f8111d1d2ceab3428f35df520623d348ffbbbd0060660c55fe3abbb68698d Oct 31 16:07:25.386909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.386879 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:25.560931 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.560848 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:25.835464 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.835390 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn"] Oct 31 16:07:25.840147 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.840117 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.893986 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.893952 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8f6jw\"" Oct 31 16:07:25.896989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.896962 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn"] Oct 31 16:07:25.907634 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907605 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.907742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907647 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122fe05-653c-4619-a231-c72eefa985b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.907789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907770 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.907831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907797 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.907831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907813 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.907886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907829 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b122fe05-653c-4619-a231-c72eefa985b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.907969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907951 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122fe05-653c-4619-a231-c72eefa985b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.908005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.907988 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s229l\" (UniqueName: \"kubernetes.io/projected/b122fe05-653c-4619-a231-c72eefa985b4-kube-api-access-s229l\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.908038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.908024 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:25.945614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:25.945582 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:26.008504 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008470 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s229l\" (UniqueName: \"kubernetes.io/projected/b122fe05-653c-4619-a231-c72eefa985b4-kube-api-access-s229l\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008513 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008635 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008664 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122fe05-653c-4619-a231-c72eefa985b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008739 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008762 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008789 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008815 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b122fe05-653c-4619-a231-c72eefa985b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008870 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122fe05-653c-4619-a231-c72eefa985b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.008958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.008920 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.009112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.009055 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.009275 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.009240 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.009314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.009271 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.009547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.009530 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b122fe05-653c-4619-a231-c72eefa985b4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.011511 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.011486 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b122fe05-653c-4619-a231-c72eefa985b4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.011670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.011650 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b122fe05-653c-4619-a231-c72eefa985b4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.065295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.065268 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s229l\" (UniqueName: \"kubernetes.io/projected/b122fe05-653c-4619-a231-c72eefa985b4-kube-api-access-s229l\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.065439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.065330 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b122fe05-653c-4619-a231-c72eefa985b4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjljn\" (UID: \"b122fe05-653c-4619-a231-c72eefa985b4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.143522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.143491 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:26.153313 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.153289 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:26.161543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.161520 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" event={"ID":"cb9fb5ff-4889-4cf7-aa3a-37a80b636932","Type":"ContainerStarted","Data":"b31f8111d1d2ceab3428f35df520623d348ffbbbd0060660c55fe3abbb68698d"} Oct 31 16:07:26.565500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.565472 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn"] Oct 31 16:07:26.567393 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:26.567355 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb122fe05_653c_4619_a231_c72eefa985b4.slice/crio-5948f3798ec47ee55df10701ea3ef2af5363520173a07355a4a11fea0c445e45 WatchSource:0}: Error finding container 5948f3798ec47ee55df10701ea3ef2af5363520173a07355a4a11fea0c445e45: Status 404 returned error can't find the container with id 5948f3798ec47ee55df10701ea3ef2af5363520173a07355a4a11fea0c445e45 Oct 31 16:07:26.569786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.569756 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:07:26.569874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.569818 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:07:26.569874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:26.569847 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:07:27.168764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:27.168728 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" event={"ID":"b122fe05-653c-4619-a231-c72eefa985b4","Type":"ContainerStarted","Data":"10aca1987461d2e4b3f253b5cf1e703423c422b95bfb9b31afea7f0aaba995b9"} Oct 31 16:07:27.168909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:27.168783 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" event={"ID":"b122fe05-653c-4619-a231-c72eefa985b4","Type":"ContainerStarted","Data":"5948f3798ec47ee55df10701ea3ef2af5363520173a07355a4a11fea0c445e45"} Oct 31 16:07:27.279358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:27.279296 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" podStartSLOduration=2.279277648 podStartE2EDuration="2.279277648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:27.277156931 +0000 UTC m=+294.312451738" watchObservedRunningTime="2026-10-31 16:07:27.279277648 +0000 UTC m=+294.314572453" Oct 31 16:07:28.131869 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.131797 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:28.153632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.153608 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:28.159042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.159020 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:28.177714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.177681 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" event={"ID":"cb9fb5ff-4889-4cf7-aa3a-37a80b636932","Type":"ContainerStarted","Data":"08e44b311dcdc5eb1a5c9b5bc0dac34f23beb2ea8be0527c6b1ad40156282fad"} Oct 31 16:07:28.177845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.177779 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:07:28.177991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.177976 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:28.178818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.178801 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjljn" Oct 31 16:07:28.278915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.278880 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:28.528548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.528491 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" podStartSLOduration=1.922362938 podStartE2EDuration="4.52847734s" totalImagesPullingTime="2.606114402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:28.527399441 +0000 UTC m=+295.562694244" watchObservedRunningTime="2026-10-31 16:07:28.52847734 +0000 UTC m=+295.563772154" Oct 31 16:07:28.535801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.535774 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:28.691100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.691064 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:28.931823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:28.931792 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:29.130047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:29.130013 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:30.357013 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:30.356976 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:30.597954 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:30.597917 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:30.740439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:30.740403 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:31.736728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:31.736692 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:31.892229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:31.892192 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:33.097186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.097153 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:07:33.097961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.097940 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:07:33.104069 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.104047 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:07:33.104850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.104831 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:07:33.115298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.115271 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:07:33.115907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.115890 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:07:33.180798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.180767 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h99s9"] Oct 31 16:07:33.189907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.189876 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" Oct 31 16:07:33.238352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.238302 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hxc6d\"" Oct 31 16:07:33.238828 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.238799 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:33.278701 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.278668 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qqf\" (UniqueName: \"kubernetes.io/projected/686d83c5-e6cb-4b0a-bf58-c23c1295b11a-kube-api-access-j2qqf\") pod \"authorino-56c6c9ddd5-h99s9\" (UID: \"686d83c5-e6cb-4b0a-bf58-c23c1295b11a\") " pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" Oct 31 16:07:33.288086 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.288054 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h99s9"] Oct 31 16:07:33.385522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.380434 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qqf\" (UniqueName: \"kubernetes.io/projected/686d83c5-e6cb-4b0a-bf58-c23c1295b11a-kube-api-access-j2qqf\") pod \"authorino-56c6c9ddd5-h99s9\" (UID: \"686d83c5-e6cb-4b0a-bf58-c23c1295b11a\") " pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" Oct 31 16:07:33.432731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.432696 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:33.433251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.433228 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qqf\" (UniqueName: \"kubernetes.io/projected/686d83c5-e6cb-4b0a-bf58-c23c1295b11a-kube-api-access-j2qqf\") pod \"authorino-56c6c9ddd5-h99s9\" (UID: \"686d83c5-e6cb-4b0a-bf58-c23c1295b11a\") " pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" Oct 31 16:07:33.501993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.501952 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" Oct 31 16:07:33.576323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.576274 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:33.866745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:33.866720 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h99s9"] Oct 31 16:07:33.868805 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:33.868768 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod686d83c5_e6cb_4b0a_bf58_c23c1295b11a.slice/crio-6b473cc280f62bdd1062e1df579b0fde6d185f426b0b3db7b0d8fcf0c101b32f WatchSource:0}: Error finding container 6b473cc280f62bdd1062e1df579b0fde6d185f426b0b3db7b0d8fcf0c101b32f: Status 404 returned error can't find the container with id 6b473cc280f62bdd1062e1df579b0fde6d185f426b0b3db7b0d8fcf0c101b32f Oct 31 16:07:33.882794 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:07:33.882761 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:07:34.210298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:34.210257 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" event={"ID":"686d83c5-e6cb-4b0a-bf58-c23c1295b11a","Type":"ContainerStarted","Data":"6b473cc280f62bdd1062e1df579b0fde6d185f426b0b3db7b0d8fcf0c101b32f"} Oct 31 16:07:35.156882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:35.156852 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:35.157058 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:35.156921 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f5vdt" Oct 31 16:07:35.203341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:35.203302 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:35.357014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:35.356976 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:37.227465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:37.227433 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" event={"ID":"686d83c5-e6cb-4b0a-bf58-c23c1295b11a","Type":"ContainerStarted","Data":"343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac"} Oct 31 16:07:37.333590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:37.333540 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" podStartSLOduration=1.596759757 podStartE2EDuration="4.333526562s" totalImagesPullingTime="2.736766805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:07:37.331568344 +0000 UTC m=+304.366863149" watchObservedRunningTime="2026-10-31 16:07:37.333526562 +0000 UTC m=+304.368821369" Oct 31 16:07:37.614706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:37.614625 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:37.826461 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:37.826428 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:39.184970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:39.184941 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:07:44.144509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:44.144433 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:44.335978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:44.335946 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:49.142532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:49.142498 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:49.294034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:49.293987 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:49.549640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:49.549557 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:49.706899 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:49.706858 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:49.960522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:49.960487 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:50.112124 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:50.112090 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:53.417170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.417104 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh"] Oct 31 16:07:53.417548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.417339 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" podUID="f00d7cbc-2929-4309-acc5-ffc348c41a85" containerName="manager" containerID="cri-o://4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d" gracePeriod=10 Oct 31 16:07:53.658938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.658917 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:53.759839 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.759776 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00d7cbc-2929-4309-acc5-ffc348c41a85-kube-api-access-q6p82\" (UniqueName: \"kubernetes.io/projected/f00d7cbc-2929-4309-acc5-ffc348c41a85-kube-api-access-q6p82\") pod \"f00d7cbc-2929-4309-acc5-ffc348c41a85\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " Oct 31 16:07:53.759995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.759839 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f00d7cbc-2929-4309-acc5-ffc348c41a85-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00d7cbc-2929-4309-acc5-ffc348c41a85-extensions-socket-volume\") pod \"f00d7cbc-2929-4309-acc5-ffc348c41a85\" (UID: \"f00d7cbc-2929-4309-acc5-ffc348c41a85\") " Oct 31 16:07:53.760286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.760262 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00d7cbc-2929-4309-acc5-ffc348c41a85-extensions-socket-volume" pod "f00d7cbc-2929-4309-acc5-ffc348c41a85" (UID: "f00d7cbc-2929-4309-acc5-ffc348c41a85"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:53.761932 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.761910 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00d7cbc-2929-4309-acc5-ffc348c41a85-kube-api-access-q6p82" pod "f00d7cbc-2929-4309-acc5-ffc348c41a85" (UID: "f00d7cbc-2929-4309-acc5-ffc348c41a85"). InnerVolumeSpecName "kube-api-access-q6p82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:53.861190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.861114 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6p82\" (UniqueName: \"kubernetes.io/projected/f00d7cbc-2929-4309-acc5-ffc348c41a85-kube-api-access-q6p82\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:07:53.861190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:53.861184 4880 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f00d7cbc-2929-4309-acc5-ffc348c41a85-extensions-socket-volume\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:07:54.319964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.319928 4880 generic.go:356] "Generic (PLEG): container finished" podID="f00d7cbc-2929-4309-acc5-ffc348c41a85" containerID="4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d" exitCode=0 Oct 31 16:07:54.320215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.319985 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" Oct 31 16:07:54.320215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.320017 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" event={"ID":"f00d7cbc-2929-4309-acc5-ffc348c41a85","Type":"ContainerDied","Data":"4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d"} Oct 31 16:07:54.320215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.320073 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh" event={"ID":"f00d7cbc-2929-4309-acc5-ffc348c41a85","Type":"ContainerDied","Data":"5d7345f3476e09379cd88284b862706e3ffa9b880c8a8dab7710f869345483ad"} Oct 31 16:07:54.320215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.320094 4880 scope.go:122] "RemoveContainer" containerID="4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d" Oct 31 16:07:54.329205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.329186 4880 scope.go:122] "RemoveContainer" containerID="4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d" Oct 31 16:07:54.329441 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:07:54.329424 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d\": container with ID starting with 4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d not found: ID does not exist" containerID="4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d" Oct 31 16:07:54.329488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.329449 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d"} err="failed to get container status \"4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d\": rpc error: code = NotFound desc = could not find container \"4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d\": container with ID starting with 4290ecd9116dcbf04bce54fe4cee48f15ff450bd07a20c2f4ae20c825caa944d not found: ID does not exist" Oct 31 16:07:54.473390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.473360 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh"] Oct 31 16:07:54.478569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.478549 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-6wxxh"] Oct 31 16:07:54.538985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.538958 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:54.749166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:54.749098 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:55.089243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:55.089172 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00d7cbc-2929-4309-acc5-ffc348c41a85" path="/var/lib/kubelet/pods/f00d7cbc-2929-4309-acc5-ffc348c41a85/volumes" Oct 31 16:07:55.634997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:55.634969 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:55.847987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:55.847947 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:56.293019 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.292987 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6"] Oct 31 16:07:56.293375 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.293362 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f00d7cbc-2929-4309-acc5-ffc348c41a85" containerName="manager" Oct 31 16:07:56.293436 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.293377 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00d7cbc-2929-4309-acc5-ffc348c41a85" containerName="manager" Oct 31 16:07:56.293471 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.293465 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00d7cbc-2929-4309-acc5-ffc348c41a85" containerName="manager" Oct 31 16:07:56.341988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.341961 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.350230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.350023 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6"] Oct 31 16:07:56.392372 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.392350 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:07:56.392372 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.392362 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:07:56.401724 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.401700 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:56.421551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.421528 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 16:07:56.437041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.437014 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-pz4tg\"" Oct 31 16:07:56.484640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.484606 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef573716-f984-4646-855d-60a8d4315097-data\") pod \"postgres-7cdbf6dc44-f7jt6\" (UID: \"ef573716-f984-4646-855d-60a8d4315097\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.484809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.484668 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7vb\" (UniqueName: \"kubernetes.io/projected/ef573716-f984-4646-855d-60a8d4315097-kube-api-access-gw7vb\") pod \"postgres-7cdbf6dc44-f7jt6\" (UID: \"ef573716-f984-4646-855d-60a8d4315097\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.568590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.568509 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:56.585356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.585327 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef573716-f984-4646-855d-60a8d4315097-data\") pod \"postgres-7cdbf6dc44-f7jt6\" (UID: \"ef573716-f984-4646-855d-60a8d4315097\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.585591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.585423 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7vb\" (UniqueName: \"kubernetes.io/projected/ef573716-f984-4646-855d-60a8d4315097-kube-api-access-gw7vb\") pod \"postgres-7cdbf6dc44-f7jt6\" (UID: \"ef573716-f984-4646-855d-60a8d4315097\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.585816 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.585793 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef573716-f984-4646-855d-60a8d4315097-data\") pod \"postgres-7cdbf6dc44-f7jt6\" (UID: \"ef573716-f984-4646-855d-60a8d4315097\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.638625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.638599 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7vb\" (UniqueName: \"kubernetes.io/projected/ef573716-f984-4646-855d-60a8d4315097-kube-api-access-gw7vb\") pod \"postgres-7cdbf6dc44-f7jt6\" (UID: \"ef573716-f984-4646-855d-60a8d4315097\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.666128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.666100 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:07:56.725594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:56.725560 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:57.048503 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:57.048475 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6"] Oct 31 16:07:57.049962 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:07:57.049937 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef573716_f984_4646_855d_60a8d4315097.slice/crio-ea9531a614bf8ad331236d1e74b267cbec76b9d3db6b095a0408b7cde782cd2a WatchSource:0}: Error finding container ea9531a614bf8ad331236d1e74b267cbec76b9d3db6b095a0408b7cde782cd2a: Status 404 returned error can't find the container with id ea9531a614bf8ad331236d1e74b267cbec76b9d3db6b095a0408b7cde782cd2a Oct 31 16:07:57.051484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:57.051465 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:07:57.337839 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:57.337758 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" event={"ID":"ef573716-f984-4646-855d-60a8d4315097","Type":"ContainerStarted","Data":"ea9531a614bf8ad331236d1e74b267cbec76b9d3db6b095a0408b7cde782cd2a"} Oct 31 16:07:57.470839 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:57.470811 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:57.617240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:57.617163 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:58.519452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:58.519413 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:58.673268 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:58.673229 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:58.812103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:58.812030 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:58.995074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:58.995035 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:07:59.156021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:07:59.155986 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:03.880061 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:08:03.880023 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:08:04.379831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:04.379779 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" event={"ID":"ef573716-f984-4646-855d-60a8d4315097","Type":"ContainerStarted","Data":"200eec7e7347c1e4fdd002ade69e9d44fff2e55e3b1b587d728ef42fe01c63f2"} Oct 31 16:08:04.380024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:04.379966 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:08:04.484904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:04.484842 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" podStartSLOduration=1.8382347270000001 podStartE2EDuration="8.484826433s" totalImagesPullingTime="6.646591706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:04.483730252 +0000 UTC m=+331.519025055" watchObservedRunningTime="2026-10-31 16:08:04.484826433 +0000 UTC m=+331.520121238" Oct 31 16:08:05.581991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:05.581954 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:05.849244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:05.849157 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:10.415681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:10.415655 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-f7jt6" Oct 31 16:08:16.272892 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:16.272858 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:16.427063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:16.427031 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:17.362776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.362742 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:17.413953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.413919 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgtfp"] Oct 31 16:08:17.417543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.417518 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.453632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.453600 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-webhook-cert\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.453761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.453650 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdtp\" (UniqueName: \"kubernetes.io/projected/e158a6d2-afdd-457d-be1b-938d7f4807b6-kube-api-access-wmdtp\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.453761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.453751 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-metrics-certs\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.511149 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.511116 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:17.512742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.512722 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v4hcf\"" Oct 31 16:08:17.512810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.512722 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 16:08:17.512810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.512742 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 16:08:17.512810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.512769 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 16:08:17.513014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.512874 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 16:08:17.513014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.512936 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 16:08:17.515477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.515454 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgtfp"] Oct 31 16:08:17.554391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.554324 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdtp\" (UniqueName: \"kubernetes.io/projected/e158a6d2-afdd-457d-be1b-938d7f4807b6-kube-api-access-wmdtp\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.554632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.554435 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-metrics-certs\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.554632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.554481 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-webhook-cert\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.557334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.557312 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-metrics-certs\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.557452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.557352 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-webhook-cert\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.607092 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.607057 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdtp\" (UniqueName: \"kubernetes.io/projected/e158a6d2-afdd-457d-be1b-938d7f4807b6-kube-api-access-wmdtp\") pod \"maas-controller-579cf7c6d-mgtfp\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.656539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.656509 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:17.730536 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.730487 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:17.809443 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.809412 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:17.820207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.820179 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c95c64d65-fsxfh"] Oct 31 16:08:17.823830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.823812 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.856900 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.856875 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2xg\" (UniqueName: \"kubernetes.io/projected/7daa1454-dc7a-41a9-9b9a-332fe155cb73-kube-api-access-dl2xg\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.856992 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.856911 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-webhook-cert\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.856992 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.856933 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-metrics-certs\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.879552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.879525 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-fsxfh"] Oct 31 16:08:17.958062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.957975 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2xg\" (UniqueName: \"kubernetes.io/projected/7daa1454-dc7a-41a9-9b9a-332fe155cb73-kube-api-access-dl2xg\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.958062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.958023 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-webhook-cert\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.958062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.958048 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-metrics-certs\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.960427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.960403 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-metrics-certs\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:17.960527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:17.960444 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-webhook-cert\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:18.010607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:18.010584 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2xg\" (UniqueName: \"kubernetes.io/projected/7daa1454-dc7a-41a9-9b9a-332fe155cb73-kube-api-access-dl2xg\") pod \"maas-controller-6c95c64d65-fsxfh\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:18.094903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:18.094883 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgtfp"] Oct 31 16:08:18.096457 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:08:18.096430 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode158a6d2_afdd_457d_be1b_938d7f4807b6.slice/crio-beab985924cbc58e8f0b9d99c570d2daae6850fae882304150be3ab98cbced77 WatchSource:0}: Error finding container beab985924cbc58e8f0b9d99c570d2daae6850fae882304150be3ab98cbced77: Status 404 returned error can't find the container with id beab985924cbc58e8f0b9d99c570d2daae6850fae882304150be3ab98cbced77 Oct 31 16:08:18.136591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:18.136565 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:18.453943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:18.453908 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" event={"ID":"e158a6d2-afdd-457d-be1b-938d7f4807b6","Type":"ContainerStarted","Data":"beab985924cbc58e8f0b9d99c570d2daae6850fae882304150be3ab98cbced77"} Oct 31 16:08:18.500247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:18.500224 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-fsxfh"] Oct 31 16:08:18.502090 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:08:18.502066 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7daa1454_dc7a_41a9_9b9a_332fe155cb73.slice/crio-55d8acb111a8bffb288ebafb518ca7e7b9219de683c1a4dc437339fbe3dfedcc WatchSource:0}: Error finding container 55d8acb111a8bffb288ebafb518ca7e7b9219de683c1a4dc437339fbe3dfedcc: Status 404 returned error can't find the container with id 55d8acb111a8bffb288ebafb518ca7e7b9219de683c1a4dc437339fbe3dfedcc Oct 31 16:08:19.460320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:19.460280 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" event={"ID":"7daa1454-dc7a-41a9-9b9a-332fe155cb73","Type":"ContainerStarted","Data":"55d8acb111a8bffb288ebafb518ca7e7b9219de683c1a4dc437339fbe3dfedcc"} Oct 31 16:08:22.261255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:22.261220 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:22.463665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:22.463603 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:22.482456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:22.482420 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" event={"ID":"e158a6d2-afdd-457d-be1b-938d7f4807b6","Type":"ContainerStarted","Data":"9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3"} Oct 31 16:08:22.482622 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:22.482495 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:22.483939 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:22.483912 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" event={"ID":"7daa1454-dc7a-41a9-9b9a-332fe155cb73","Type":"ContainerStarted","Data":"e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827"} Oct 31 16:08:22.484077 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:22.483965 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:23.102094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:23.102041 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" podStartSLOduration=2.333615279 podStartE2EDuration="6.102026843s" totalImagesPullingTime="3.768411564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:23.099485666 +0000 UTC m=+350.134780470" watchObservedRunningTime="2026-10-31 16:08:23.102026843 +0000 UTC m=+350.137321647" Oct 31 16:08:23.234272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:23.234218 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" podStartSLOduration=2.790324799 podStartE2EDuration="6.234203565s" totalImagesPullingTime="3.443878766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:23.234051712 +0000 UTC m=+350.269346518" watchObservedRunningTime="2026-10-31 16:08:23.234203565 +0000 UTC m=+350.269498371" Oct 31 16:08:25.094527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:25.094496 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:29.037754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.037720 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9"] Oct 31 16:08:29.041504 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.041485 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.093421 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.093391 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9"] Oct 31 16:08:29.104811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.104781 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 16:08:29.104919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.104824 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 16:08:29.151191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.151162 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 16:08:29.151430 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.151412 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-8b26r\"" Oct 31 16:08:29.157938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.157911 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqhz\" (UniqueName: \"kubernetes.io/projected/226b0933-6b0f-471a-a4cf-be67d2fbed67-kube-api-access-ldqhz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.158034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.157945 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.158034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.157972 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/226b0933-6b0f-471a-a4cf-be67d2fbed67-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.158126 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.158106 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.212929 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.212900 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:29.258840 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.258813 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.258970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.258897 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqhz\" (UniqueName: \"kubernetes.io/projected/226b0933-6b0f-471a-a4cf-be67d2fbed67-kube-api-access-ldqhz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.258970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.258914 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.258970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.258941 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/226b0933-6b0f-471a-a4cf-be67d2fbed67-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.259078 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:08:29.259028 4880 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 31 16:08:29.259111 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:08:29.259098 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-proxy-source podName:226b0933-6b0f-471a-a4cf-be67d2fbed67 nodeName:}" failed. No retries permitted until 2026-10-31 16:08:29.759079359 +0000 UTC m=+356.794374154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" (UID: "226b0933-6b0f-471a-a4cf-be67d2fbed67") : configmap "usage-tenancy-proxy-source" not found Oct 31 16:08:29.259182 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:08:29.259034 4880 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 31 16:08:29.259182 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:08:29.259178 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226b0933-6b0f-471a-a4cf-be67d2fbed67-tls-cert podName:226b0933-6b0f-471a-a4cf-be67d2fbed67 nodeName:}" failed. No retries permitted until 2026-10-31 16:08:29.759161707 +0000 UTC m=+356.794456496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/226b0933-6b0f-471a-a4cf-be67d2fbed67-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" (UID: "226b0933-6b0f-471a-a4cf-be67d2fbed67") : secret "usage-logs-tenancy-proxy-tls" not found Oct 31 16:08:29.259521 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.259505 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.328217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.328091 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqhz\" (UniqueName: \"kubernetes.io/projected/226b0933-6b0f-471a-a4cf-be67d2fbed67-kube-api-access-ldqhz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.428088 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.428058 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:29.763678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.763643 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.763853 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.763687 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/226b0933-6b0f-471a-a4cf-be67d2fbed67-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.764570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.764548 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/226b0933-6b0f-471a-a4cf-be67d2fbed67-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.766067 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.766048 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/226b0933-6b0f-471a-a4cf-be67d2fbed67-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6qjm9\" (UID: \"226b0933-6b0f-471a-a4cf-be67d2fbed67\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:29.954187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:29.954153 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:30.338105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:30.338083 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9"] Oct 31 16:08:30.340340 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:08:30.340315 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226b0933_6b0f_471a_a4cf_be67d2fbed67.slice/crio-f23965c4ba965b3fa9fde813ae80e424bd44f1e499cb979e8d36457abe89ca37 WatchSource:0}: Error finding container f23965c4ba965b3fa9fde813ae80e424bd44f1e499cb979e8d36457abe89ca37: Status 404 returned error can't find the container with id f23965c4ba965b3fa9fde813ae80e424bd44f1e499cb979e8d36457abe89ca37 Oct 31 16:08:30.527797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:30.527762 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" event={"ID":"226b0933-6b0f-471a-a4cf-be67d2fbed67","Type":"ContainerStarted","Data":"f23965c4ba965b3fa9fde813ae80e424bd44f1e499cb979e8d36457abe89ca37"} Oct 31 16:08:33.491797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.491720 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:08:33.491797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.491795 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:33.680038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.680002 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgtfp"] Oct 31 16:08:33.680213 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.680199 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" podUID="e158a6d2-afdd-457d-be1b-938d7f4807b6" containerName="manager" containerID="cri-o://9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3" gracePeriod=30 Oct 31 16:08:33.885442 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:08:33.885402 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:08:33.930978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.930929 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:33.997155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.997095 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e158a6d2-afdd-457d-be1b-938d7f4807b6-kube-api-access-wmdtp\" (UniqueName: \"kubernetes.io/projected/e158a6d2-afdd-457d-be1b-938d7f4807b6-kube-api-access-wmdtp\") pod \"e158a6d2-afdd-457d-be1b-938d7f4807b6\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " Oct 31 16:08:33.997311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.997180 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-metrics-certs\") pod \"e158a6d2-afdd-457d-be1b-938d7f4807b6\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " Oct 31 16:08:33.997311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.997227 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-webhook-cert\") pod \"e158a6d2-afdd-457d-be1b-938d7f4807b6\" (UID: \"e158a6d2-afdd-457d-be1b-938d7f4807b6\") " Oct 31 16:08:33.999355 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.999320 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e158a6d2-afdd-457d-be1b-938d7f4807b6-kube-api-access-wmdtp" pod "e158a6d2-afdd-457d-be1b-938d7f4807b6" (UID: "e158a6d2-afdd-457d-be1b-938d7f4807b6"). InnerVolumeSpecName "kube-api-access-wmdtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:08:33.999355 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.999339 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-metrics-certs" pod "e158a6d2-afdd-457d-be1b-938d7f4807b6" (UID: "e158a6d2-afdd-457d-be1b-938d7f4807b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:08:33.999495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:33.999407 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-webhook-cert" pod "e158a6d2-afdd-457d-be1b-938d7f4807b6" (UID: "e158a6d2-afdd-457d-be1b-938d7f4807b6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:08:34.098417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.098346 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmdtp\" (UniqueName: \"kubernetes.io/projected/e158a6d2-afdd-457d-be1b-938d7f4807b6-kube-api-access-wmdtp\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:08:34.098417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.098372 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:08:34.098417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.098381 4880 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e158a6d2-afdd-457d-be1b-938d7f4807b6-webhook-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:08:34.560309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.560267 4880 generic.go:356] "Generic (PLEG): container finished" podID="e158a6d2-afdd-457d-be1b-938d7f4807b6" containerID="9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3" exitCode=0 Oct 31 16:08:34.560926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.560365 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" event={"ID":"e158a6d2-afdd-457d-be1b-938d7f4807b6","Type":"ContainerDied","Data":"9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3"} Oct 31 16:08:34.560926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.560417 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" event={"ID":"e158a6d2-afdd-457d-be1b-938d7f4807b6","Type":"ContainerDied","Data":"beab985924cbc58e8f0b9d99c570d2daae6850fae882304150be3ab98cbced77"} Oct 31 16:08:34.560926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.560447 4880 scope.go:122] "RemoveContainer" containerID="9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3" Oct 31 16:08:34.560926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.560612 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-mgtfp" Oct 31 16:08:34.576919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.576856 4880 scope.go:122] "RemoveContainer" containerID="9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3" Oct 31 16:08:34.577470 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:08:34.577425 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3\": container with ID starting with 9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3 not found: ID does not exist" containerID="9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3" Oct 31 16:08:34.577575 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.577465 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3"} err="failed to get container status \"9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3\": rpc error: code = NotFound desc = could not find container \"9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3\": container with ID starting with 9e56ebb0eb69882d69d407d1830aa1f44f90dddf83fc9b5e8e071a24b003a8a3 not found: ID does not exist" Oct 31 16:08:34.746888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.746836 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgtfp"] Oct 31 16:08:34.799087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:34.799017 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-mgtfp"] Oct 31 16:08:35.092330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:35.091939 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e158a6d2-afdd-457d-be1b-938d7f4807b6" path="/var/lib/kubelet/pods/e158a6d2-afdd-457d-be1b-938d7f4807b6/volumes" Oct 31 16:08:43.624007 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:43.623973 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" event={"ID":"226b0933-6b0f-471a-a4cf-be67d2fbed67","Type":"ContainerStarted","Data":"df7e16171b48cdd2a23fd67a64473af935b20ad0b6563d1cd56892659f178410"} Oct 31 16:08:43.624413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:43.624126 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:43.750212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:43.750087 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" podStartSLOduration=2.638943936 podStartE2EDuration="15.750074152s" totalImagesPullingTime="13.111130216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:43.748891816 +0000 UTC m=+370.784186621" watchObservedRunningTime="2026-10-31 16:08:43.750074152 +0000 UTC m=+370.785368956" Oct 31 16:08:46.318872 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:46.318837 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:46.470419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:46.470387 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:46.629082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:46.628991 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:46.780983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:46.780942 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:46.928651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:46.928614 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:47.080544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.080505 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:47.249400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.249319 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:47.256636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.256611 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9l8zz"] Oct 31 16:08:47.256985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.256974 4880 cpu_manager.go:414] "containerMap: removing container" podUID="e158a6d2-afdd-457d-be1b-938d7f4807b6" containerName="manager" Oct 31 16:08:47.257032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.256987 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e158a6d2-afdd-457d-be1b-938d7f4807b6" containerName="manager" Oct 31 16:08:47.257069 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.257064 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="e158a6d2-afdd-457d-be1b-938d7f4807b6" containerName="manager" Oct 31 16:08:47.259931 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.259915 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.312210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.312182 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9l8zz"] Oct 31 16:08:47.325192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.325119 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbb9m\" (UniqueName: \"kubernetes.io/projected/27b75c60-a73e-4281-ace2-d2fa046c2f6d-kube-api-access-xbb9m\") pod \"kserve-controller-manager-776cc66d47-9l8zz\" (UID: \"27b75c60-a73e-4281-ace2-d2fa046c2f6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.325638 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.325251 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b75c60-a73e-4281-ace2-d2fa046c2f6d-cert\") pod \"kserve-controller-manager-776cc66d47-9l8zz\" (UID: \"27b75c60-a73e-4281-ace2-d2fa046c2f6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.365394 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.365371 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xpzjt\"" Oct 31 16:08:47.365394 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.365371 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 16:08:47.378876 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.378848 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5"] Oct 31 16:08:47.382107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.382090 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.405524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.405496 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:47.426356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.426317 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbb9m\" (UniqueName: \"kubernetes.io/projected/27b75c60-a73e-4281-ace2-d2fa046c2f6d-kube-api-access-xbb9m\") pod \"kserve-controller-manager-776cc66d47-9l8zz\" (UID: \"27b75c60-a73e-4281-ace2-d2fa046c2f6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.426487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.426397 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b75c60-a73e-4281-ace2-d2fa046c2f6d-cert\") pod \"kserve-controller-manager-776cc66d47-9l8zz\" (UID: \"27b75c60-a73e-4281-ace2-d2fa046c2f6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.429212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.429178 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b75c60-a73e-4281-ace2-d2fa046c2f6d-cert\") pod \"kserve-controller-manager-776cc66d47-9l8zz\" (UID: \"27b75c60-a73e-4281-ace2-d2fa046c2f6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.434247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.434223 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5"] Oct 31 16:08:47.439865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.439847 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-94n88\"" Oct 31 16:08:47.439988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.439970 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 16:08:47.490940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.490911 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbb9m\" (UniqueName: \"kubernetes.io/projected/27b75c60-a73e-4281-ace2-d2fa046c2f6d-kube-api-access-xbb9m\") pod \"kserve-controller-manager-776cc66d47-9l8zz\" (UID: \"27b75c60-a73e-4281-ace2-d2fa046c2f6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.527025 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.526945 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbm9\" (UniqueName: \"kubernetes.io/projected/316dd107-ce44-4a87-8627-70c9620b349b-kube-api-access-8zbm9\") pod \"llmisvc-controller-manager-869d5dcc5d-5cjz5\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.527025 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.526985 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316dd107-ce44-4a87-8627-70c9620b349b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5cjz5\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.563060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.563024 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:47.572041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.572021 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:47.623760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.623723 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ld2rc"] Oct 31 16:08:47.628266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.628247 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.628693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.628665 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbm9\" (UniqueName: \"kubernetes.io/projected/316dd107-ce44-4a87-8627-70c9620b349b-kube-api-access-8zbm9\") pod \"llmisvc-controller-manager-869d5dcc5d-5cjz5\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.628856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.628836 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316dd107-ce44-4a87-8627-70c9620b349b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5cjz5\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.635047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.635005 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316dd107-ce44-4a87-8627-70c9620b349b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5cjz5\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.679586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.679554 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ld2rc"] Oct 31 16:08:47.686831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.686807 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 16:08:47.686972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.686841 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 16:08:47.686972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.686899 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-58bqj\"" Oct 31 16:08:47.691394 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.691372 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbm9\" (UniqueName: \"kubernetes.io/projected/316dd107-ce44-4a87-8627-70c9620b349b-kube-api-access-8zbm9\") pod \"llmisvc-controller-manager-869d5dcc5d-5cjz5\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.700228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.700207 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:47.730108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.730065 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2b1a4b7-4b50-45ae-a68e-af611a888a54-cert\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.730301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.730201 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxzt\" (UniqueName: \"kubernetes.io/projected/e2b1a4b7-4b50-45ae-a68e-af611a888a54-kube-api-access-jnxzt\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.730375 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.730305 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2b1a4b7-4b50-45ae-a68e-af611a888a54-metrics-certs\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.830884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.830800 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2b1a4b7-4b50-45ae-a68e-af611a888a54-metrics-certs\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.830884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.830856 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2b1a4b7-4b50-45ae-a68e-af611a888a54-cert\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.831110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.830907 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxzt\" (UniqueName: \"kubernetes.io/projected/e2b1a4b7-4b50-45ae-a68e-af611a888a54-kube-api-access-jnxzt\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.833287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.833265 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2b1a4b7-4b50-45ae-a68e-af611a888a54-metrics-certs\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.833365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.833277 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2b1a4b7-4b50-45ae-a68e-af611a888a54-cert\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.892734 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.892702 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxzt\" (UniqueName: \"kubernetes.io/projected/e2b1a4b7-4b50-45ae-a68e-af611a888a54-kube-api-access-jnxzt\") pod \"odh-model-controller-769875db56-ld2rc\" (UID: \"e2b1a4b7-4b50-45ae-a68e-af611a888a54\") " pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.954565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.954530 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:47.958007 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:47.957976 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9l8zz"] Oct 31 16:08:47.959674 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:08:47.959649 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b75c60_a73e_4281_ace2_d2fa046c2f6d.slice/crio-5f6eb472938601d71219dd06d5ffbdff987decd7f4381ba125ddc528c47b6320 WatchSource:0}: Error finding container 5f6eb472938601d71219dd06d5ffbdff987decd7f4381ba125ddc528c47b6320: Status 404 returned error can't find the container with id 5f6eb472938601d71219dd06d5ffbdff987decd7f4381ba125ddc528c47b6320 Oct 31 16:08:48.083149 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:48.083116 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5"] Oct 31 16:08:48.086220 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:08:48.086190 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod316dd107_ce44_4a87_8627_70c9620b349b.slice/crio-88fd9768c81ccb93f0e06687f6b809a41eb8deea8434a9efc724566ee7c095bd WatchSource:0}: Error finding container 88fd9768c81ccb93f0e06687f6b809a41eb8deea8434a9efc724566ee7c095bd: Status 404 returned error can't find the container with id 88fd9768c81ccb93f0e06687f6b809a41eb8deea8434a9efc724566ee7c095bd Oct 31 16:08:48.338075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:48.338049 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ld2rc"] Oct 31 16:08:48.339370 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:08:48.339346 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b1a4b7_4b50_45ae_a68e_af611a888a54.slice/crio-13f9f42c0f6abc0d386b98352d6edc12ec48807ad4a6d5c93e0d79598c41a19d WatchSource:0}: Error finding container 13f9f42c0f6abc0d386b98352d6edc12ec48807ad4a6d5c93e0d79598c41a19d: Status 404 returned error can't find the container with id 13f9f42c0f6abc0d386b98352d6edc12ec48807ad4a6d5c93e0d79598c41a19d Oct 31 16:08:48.652861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:48.652830 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ld2rc" event={"ID":"e2b1a4b7-4b50-45ae-a68e-af611a888a54","Type":"ContainerStarted","Data":"13f9f42c0f6abc0d386b98352d6edc12ec48807ad4a6d5c93e0d79598c41a19d"} Oct 31 16:08:48.654105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:48.654079 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" event={"ID":"316dd107-ce44-4a87-8627-70c9620b349b","Type":"ContainerStarted","Data":"88fd9768c81ccb93f0e06687f6b809a41eb8deea8434a9efc724566ee7c095bd"} Oct 31 16:08:48.655158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:48.655125 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" event={"ID":"27b75c60-a73e-4281-ace2-d2fa046c2f6d","Type":"ContainerStarted","Data":"5f6eb472938601d71219dd06d5ffbdff987decd7f4381ba125ddc528c47b6320"} Oct 31 16:08:48.754857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:48.754819 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:52.686218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.686180 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" event={"ID":"316dd107-ce44-4a87-8627-70c9620b349b","Type":"ContainerStarted","Data":"1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b"} Oct 31 16:08:52.686660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.686379 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:08:52.687922 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.687894 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" event={"ID":"27b75c60-a73e-4281-ace2-d2fa046c2f6d","Type":"ContainerStarted","Data":"ae1218938a9354798a243ec91ad2e114d20b867e84d17f0108b9cfead98afea4"} Oct 31 16:08:52.688009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.687947 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:08:52.689330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.689310 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ld2rc" event={"ID":"e2b1a4b7-4b50-45ae-a68e-af611a888a54","Type":"ContainerStarted","Data":"7b8ddcf4b360e823412ca37cf10c77016117640d0cc7d6007ec800c6f93d8f98"} Oct 31 16:08:52.689436 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.689425 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:08:52.815269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.815186 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" podStartSLOduration=1.7533569020000002 podStartE2EDuration="5.815174645s" totalImagesPullingTime="4.061817743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:52.813208995 +0000 UTC m=+379.848503803" watchObservedRunningTime="2026-10-31 16:08:52.815174645 +0000 UTC m=+379.850469448" Oct 31 16:08:52.938634 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:52.938582 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" podStartSLOduration=1.629471361 podStartE2EDuration="5.938565871s" totalImagesPullingTime="4.30909451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:52.937525763 +0000 UTC m=+379.972820567" watchObservedRunningTime="2026-10-31 16:08:52.938565871 +0000 UTC m=+379.973860667" Oct 31 16:08:53.063976 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:53.063926 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ld2rc" podStartSLOduration=1.93863528 podStartE2EDuration="6.063910478s" totalImagesPullingTime="4.125275198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:08:53.06243336 +0000 UTC m=+380.097728165" watchObservedRunningTime="2026-10-31 16:08:53.063910478 +0000 UTC m=+380.099205281" Oct 31 16:08:54.195159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:54.195086 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:54.432496 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:54.432450 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:54.596166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:54.596068 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:54.635728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:54.635701 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6qjm9" Oct 31 16:08:54.752370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:54.752328 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:54.907509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:54.907473 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:55.132502 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:55.132471 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:55.332442 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:55.332374 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:55.492961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:55.492910 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:56.016547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:56.016509 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:57.202746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:57.202700 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:57.522507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:57.522418 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:59.286474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:59.286444 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:59.495168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:59.495119 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:59.652828 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:59.652796 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:08:59.805111 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:08:59.805070 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:03.696668 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:03.696636 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ld2rc" Oct 31 16:09:03.880364 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:09:03.880328 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:09:05.747383 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:05.747347 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:12.058998 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.058945 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:12.314733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.314695 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:12.615242 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.615159 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9"] Oct 31 16:09:12.771567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.771481 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9"] Oct 31 16:09:12.771567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.771516 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.807286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.807250 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5547b96847-qs8ww"] Oct 31 16:09:12.822968 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.822937 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5547b96847-qs8ww" Oct 31 16:09:12.874987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.874904 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-metrics-certs\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.874987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.874944 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-maas-api-tls\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.875226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.875056 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m46f\" (UniqueName: \"kubernetes.io/projected/6011e6c0-bc37-4a57-b855-1a2b8a98670b-kube-api-access-8m46f\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.875226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.875194 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqk4n\" (UniqueName: \"kubernetes.io/projected/e791c8a6-201b-43a4-a3e9-e81ac1bf3083-kube-api-access-nqk4n\") pod \"authorino-5547b96847-qs8ww\" (UID: \"e791c8a6-201b-43a4-a3e9-e81ac1bf3083\") " pod="kuadrant-system/authorino-5547b96847-qs8ww" Oct 31 16:09:12.875317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.875234 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6011e6c0-bc37-4a57-b855-1a2b8a98670b-openshift-service-ca\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.884855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.884831 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7ck5g\"" Oct 31 16:09:12.884855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.884851 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 16:09:12.885021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.884960 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 16:09:12.887778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.887762 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5547b96847-qs8ww"] Oct 31 16:09:12.975820 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.975785 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m46f\" (UniqueName: \"kubernetes.io/projected/6011e6c0-bc37-4a57-b855-1a2b8a98670b-kube-api-access-8m46f\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.975988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.975848 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqk4n\" (UniqueName: \"kubernetes.io/projected/e791c8a6-201b-43a4-a3e9-e81ac1bf3083-kube-api-access-nqk4n\") pod \"authorino-5547b96847-qs8ww\" (UID: \"e791c8a6-201b-43a4-a3e9-e81ac1bf3083\") " pod="kuadrant-system/authorino-5547b96847-qs8ww" Oct 31 16:09:12.975988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.975884 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6011e6c0-bc37-4a57-b855-1a2b8a98670b-openshift-service-ca\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.975988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.975938 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-metrics-certs\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.975988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.975978 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-maas-api-tls\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.976556 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.976524 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6011e6c0-bc37-4a57-b855-1a2b8a98670b-openshift-service-ca\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.978625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.978593 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-maas-api-tls\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:12.978724 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:12.978593 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-metrics-certs\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:13.038126 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.038097 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqk4n\" (UniqueName: \"kubernetes.io/projected/e791c8a6-201b-43a4-a3e9-e81ac1bf3083-kube-api-access-nqk4n\") pod \"authorino-5547b96847-qs8ww\" (UID: \"e791c8a6-201b-43a4-a3e9-e81ac1bf3083\") " pod="kuadrant-system/authorino-5547b96847-qs8ww" Oct 31 16:09:13.042130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.042103 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m46f\" (UniqueName: \"kubernetes.io/projected/6011e6c0-bc37-4a57-b855-1a2b8a98670b-kube-api-access-8m46f\") pod \"maas-api-7bfdbbc5d5-l6xq9\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:13.112538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.112502 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:13.161300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.161275 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5547b96847-qs8ww" Oct 31 16:09:13.593183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.593123 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9"] Oct 31 16:09:13.595463 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:09:13.595437 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6011e6c0_bc37_4a57_b855_1a2b8a98670b.slice/crio-2656d3e8cf88d7aaaae1f8480724b48e7e08f018d7640d0a9d36baafa2aec62c WatchSource:0}: Error finding container 2656d3e8cf88d7aaaae1f8480724b48e7e08f018d7640d0a9d36baafa2aec62c: Status 404 returned error can't find the container with id 2656d3e8cf88d7aaaae1f8480724b48e7e08f018d7640d0a9d36baafa2aec62c Oct 31 16:09:13.595556 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.595481 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5547b96847-qs8ww"] Oct 31 16:09:13.642541 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:09:13.642509 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode791c8a6_201b_43a4_a3e9_e81ac1bf3083.slice/crio-c23be0cfbeaf98be954f020dfd63f0ca322d41d8911b553e52dc7dc4ea8ebfad WatchSource:0}: Error finding container c23be0cfbeaf98be954f020dfd63f0ca322d41d8911b553e52dc7dc4ea8ebfad: Status 404 returned error can't find the container with id c23be0cfbeaf98be954f020dfd63f0ca322d41d8911b553e52dc7dc4ea8ebfad Oct 31 16:09:13.814176 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.814121 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5547b96847-qs8ww" event={"ID":"e791c8a6-201b-43a4-a3e9-e81ac1bf3083","Type":"ContainerStarted","Data":"c23be0cfbeaf98be954f020dfd63f0ca322d41d8911b553e52dc7dc4ea8ebfad"} Oct 31 16:09:13.815473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:13.815444 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" event={"ID":"6011e6c0-bc37-4a57-b855-1a2b8a98670b","Type":"ContainerStarted","Data":"2656d3e8cf88d7aaaae1f8480724b48e7e08f018d7640d0a9d36baafa2aec62c"} Oct 31 16:09:14.839034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:14.838999 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5547b96847-qs8ww"] Oct 31 16:09:15.279794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.279760 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:54988->10.132.0.34:15021: read: connection reset by peer" start-of-body= Oct 31 16:09:15.279923 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.279841 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": read tcp 10.132.0.2:54988->10.132.0.34:15021: read: connection reset by peer" Oct 31 16:09:15.833022 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.832981 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5547b96847-qs8ww" event={"ID":"e791c8a6-201b-43a4-a3e9-e81ac1bf3083","Type":"ContainerStarted","Data":"724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6"} Oct 31 16:09:15.833302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.833039 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5547b96847-qs8ww" podUID="e791c8a6-201b-43a4-a3e9-e81ac1bf3083" containerName="authorino" containerID="cri-o://724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6" gracePeriod=30 Oct 31 16:09:15.834720 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.834697 4880 generic.go:356] "Generic (PLEG): container finished" podID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerID="6517f3ecb76d507381ad3ecba39953738e6591345f5ce5d3c1348b25c2d0b19e" exitCode=0 Oct 31 16:09:15.834801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.834775 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" event={"ID":"363aa6f3-aaba-4371-b953-557fc18a8d3f","Type":"ContainerDied","Data":"6517f3ecb76d507381ad3ecba39953738e6591345f5ce5d3c1348b25c2d0b19e"} Oct 31 16:09:15.835104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.835091 4880 scope.go:122] "RemoveContainer" containerID="6517f3ecb76d507381ad3ecba39953738e6591345f5ce5d3c1348b25c2d0b19e" Oct 31 16:09:15.899351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.899310 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:09:15.899687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.899387 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:09:15.899687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.899421 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:09:15.957870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:15.957817 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5547b96847-qs8ww" podStartSLOduration=2.358065698 podStartE2EDuration="3.957802779s" totalImagesPullingTime="1.599737081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:15.95569794 +0000 UTC m=+402.990992745" watchObservedRunningTime="2026-10-31 16:09:15.957802779 +0000 UTC m=+402.993097584" Oct 31 16:09:16.152705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.152678 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5547b96847-qs8ww" Oct 31 16:09:16.203498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.203462 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e791c8a6-201b-43a4-a3e9-e81ac1bf3083-kube-api-access-nqk4n\" (UniqueName: \"kubernetes.io/projected/e791c8a6-201b-43a4-a3e9-e81ac1bf3083-kube-api-access-nqk4n\") pod \"e791c8a6-201b-43a4-a3e9-e81ac1bf3083\" (UID: \"e791c8a6-201b-43a4-a3e9-e81ac1bf3083\") " Oct 31 16:09:16.205378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.205352 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e791c8a6-201b-43a4-a3e9-e81ac1bf3083-kube-api-access-nqk4n" pod "e791c8a6-201b-43a4-a3e9-e81ac1bf3083" (UID: "e791c8a6-201b-43a4-a3e9-e81ac1bf3083"). InnerVolumeSpecName "kube-api-access-nqk4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:16.304648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.304618 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqk4n\" (UniqueName: \"kubernetes.io/projected/e791c8a6-201b-43a4-a3e9-e81ac1bf3083-kube-api-access-nqk4n\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.850571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.850535 4880 generic.go:356] "Generic (PLEG): container finished" podID="e791c8a6-201b-43a4-a3e9-e81ac1bf3083" containerID="724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6" exitCode=0 Oct 31 16:09:16.850799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.850579 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5547b96847-qs8ww" Oct 31 16:09:16.850799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.850619 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5547b96847-qs8ww" event={"ID":"e791c8a6-201b-43a4-a3e9-e81ac1bf3083","Type":"ContainerDied","Data":"724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6"} Oct 31 16:09:16.850799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.850668 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5547b96847-qs8ww" event={"ID":"e791c8a6-201b-43a4-a3e9-e81ac1bf3083","Type":"ContainerDied","Data":"c23be0cfbeaf98be954f020dfd63f0ca322d41d8911b553e52dc7dc4ea8ebfad"} Oct 31 16:09:16.850799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.850686 4880 scope.go:122] "RemoveContainer" containerID="724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6" Oct 31 16:09:16.852772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.852749 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" event={"ID":"363aa6f3-aaba-4371-b953-557fc18a8d3f","Type":"ContainerStarted","Data":"e4dcc8f127a8c6d0ab1ee12356190c18b54d1161b2762f7e49342a6bdc22102e"} Oct 31 16:09:16.854553 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.854526 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" event={"ID":"6011e6c0-bc37-4a57-b855-1a2b8a98670b","Type":"ContainerStarted","Data":"aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e"} Oct 31 16:09:16.854658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.854583 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:16.868285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.868264 4880 scope.go:122] "RemoveContainer" containerID="724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6" Oct 31 16:09:16.868554 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:09:16.868535 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6\": container with ID starting with 724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6 not found: ID does not exist" containerID="724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6" Oct 31 16:09:16.868611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.868563 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6"} err="failed to get container status \"724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6\": rpc error: code = NotFound desc = could not find container \"724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6\": container with ID starting with 724bd9a165360d5e5f7d0248e1f8605a9db1511c72ff129a31e826e1c6e72fd6 not found: ID does not exist" Oct 31 16:09:16.887208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.887185 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:09:16.887419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.887396 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 16:09:16.887502 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:16.887450 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 16:09:17.258393 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.258319 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" podStartSLOduration=2.936949611 podStartE2EDuration="5.258296877s" totalImagesPullingTime="2.321347266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:17.184377151 +0000 UTC m=+404.219671956" watchObservedRunningTime="2026-10-31 16:09:17.258296877 +0000 UTC m=+404.293592000" Oct 31 16:09:17.260536 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.260511 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h99s9"] Oct 31 16:09:17.260724 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.260705 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" podUID="686d83c5-e6cb-4b0a-bf58-c23c1295b11a" containerName="authorino" containerID="cri-o://343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac" gracePeriod=30 Oct 31 16:09:17.310359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.310313 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5547b96847-qs8ww"] Oct 31 16:09:17.313965 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.313941 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5547b96847-qs8ww"] Oct 31 16:09:17.569335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.569313 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" Oct 31 16:09:17.616866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.616822 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/686d83c5-e6cb-4b0a-bf58-c23c1295b11a-kube-api-access-j2qqf\" (UniqueName: \"kubernetes.io/projected/686d83c5-e6cb-4b0a-bf58-c23c1295b11a-kube-api-access-j2qqf\") pod \"686d83c5-e6cb-4b0a-bf58-c23c1295b11a\" (UID: \"686d83c5-e6cb-4b0a-bf58-c23c1295b11a\") " Oct 31 16:09:17.618925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.618898 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686d83c5-e6cb-4b0a-bf58-c23c1295b11a-kube-api-access-j2qqf" pod "686d83c5-e6cb-4b0a-bf58-c23c1295b11a" (UID: "686d83c5-e6cb-4b0a-bf58-c23c1295b11a"). InnerVolumeSpecName "kube-api-access-j2qqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:17.718114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.718082 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qqf\" (UniqueName: \"kubernetes.io/projected/686d83c5-e6cb-4b0a-bf58-c23c1295b11a-kube-api-access-j2qqf\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:09:17.862483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.862394 4880 generic.go:356] "Generic (PLEG): container finished" podID="686d83c5-e6cb-4b0a-bf58-c23c1295b11a" containerID="343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac" exitCode=0 Oct 31 16:09:17.862483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.862441 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" Oct 31 16:09:17.862690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.862479 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" event={"ID":"686d83c5-e6cb-4b0a-bf58-c23c1295b11a","Type":"ContainerDied","Data":"343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac"} Oct 31 16:09:17.862690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.862527 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h99s9" event={"ID":"686d83c5-e6cb-4b0a-bf58-c23c1295b11a","Type":"ContainerDied","Data":"6b473cc280f62bdd1062e1df579b0fde6d185f426b0b3db7b0d8fcf0c101b32f"} Oct 31 16:09:17.862690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.862549 4880 scope.go:122] "RemoveContainer" containerID="343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac" Oct 31 16:09:17.872222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.872204 4880 scope.go:122] "RemoveContainer" containerID="343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac" Oct 31 16:09:17.872506 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:09:17.872484 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac\": container with ID starting with 343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac not found: ID does not exist" containerID="343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac" Oct 31 16:09:17.872594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.872519 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac"} err="failed to get container status \"343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac\": rpc error: code = NotFound desc = could not find container \"343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac\": container with ID starting with 343bc6191193a857ee74f7b80f6b77b8798efc95aef4ff5ec0cbb54d3df895ac not found: ID does not exist" Oct 31 16:09:17.887881 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.887861 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 16:09:17.887964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:17.887907 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 16:09:18.046510 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:18.046473 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h99s9"] Oct 31 16:09:18.048855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:18.048828 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h99s9"] Oct 31 16:09:18.888360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:18.888327 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 16:09:18.888751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:18.888387 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 16:09:19.088540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:19.088510 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="686d83c5-e6cb-4b0a-bf58-c23c1295b11a" path="/var/lib/kubelet/pods/686d83c5-e6cb-4b0a-bf58-c23c1295b11a/volumes" Oct 31 16:09:19.088837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:19.088825 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e791c8a6-201b-43a4-a3e9-e81ac1bf3083" path="/var/lib/kubelet/pods/e791c8a6-201b-43a4-a3e9-e81ac1bf3083/volumes" Oct 31 16:09:19.888313 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:19.888281 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 16:09:19.888477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:19.888341 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 16:09:20.887726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:20.887683 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 16:09:20.887883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:20.887738 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 16:09:21.915818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:21.915785 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:09:21.916293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:21.916166 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:09:21.917076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:21.917058 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" Oct 31 16:09:22.868019 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:22.867987 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 16:09:23.696608 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:23.696579 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:09:23.699680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:23.699660 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9l8zz" Oct 31 16:09:25.415065 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:25.415028 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:33.879934 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:09:33.879889 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:09:36.403093 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:36.403062 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:09:58.530978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.530940 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s9485"] Oct 31 16:09:58.531396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.531344 4880 cpu_manager.go:414] "containerMap: removing container" podUID="686d83c5-e6cb-4b0a-bf58-c23c1295b11a" containerName="authorino" Oct 31 16:09:58.531396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.531355 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="686d83c5-e6cb-4b0a-bf58-c23c1295b11a" containerName="authorino" Oct 31 16:09:58.531396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.531377 4880 cpu_manager.go:414] "containerMap: removing container" podUID="e791c8a6-201b-43a4-a3e9-e81ac1bf3083" containerName="authorino" Oct 31 16:09:58.531396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.531382 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e791c8a6-201b-43a4-a3e9-e81ac1bf3083" containerName="authorino" Oct 31 16:09:58.531527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.531449 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="686d83c5-e6cb-4b0a-bf58-c23c1295b11a" containerName="authorino" Oct 31 16:09:58.531527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.531459 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="e791c8a6-201b-43a4-a3e9-e81ac1bf3083" containerName="authorino" Oct 31 16:09:58.536400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.536383 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:58.588289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.588260 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s9485"] Oct 31 16:09:58.600703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.600677 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c064c108-1022-402c-8c87-761962ef7747-cert\") pod \"llmisvc-controller-manager-6466c554d8-s9485\" (UID: \"c064c108-1022-402c-8c87-761962ef7747\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:58.600810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.600781 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgbkq\" (UniqueName: \"kubernetes.io/projected/c064c108-1022-402c-8c87-761962ef7747-kube-api-access-fgbkq\") pod \"llmisvc-controller-manager-6466c554d8-s9485\" (UID: \"c064c108-1022-402c-8c87-761962ef7747\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:58.702086 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.702051 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgbkq\" (UniqueName: \"kubernetes.io/projected/c064c108-1022-402c-8c87-761962ef7747-kube-api-access-fgbkq\") pod \"llmisvc-controller-manager-6466c554d8-s9485\" (UID: \"c064c108-1022-402c-8c87-761962ef7747\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:58.702253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.702195 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c064c108-1022-402c-8c87-761962ef7747-cert\") pod \"llmisvc-controller-manager-6466c554d8-s9485\" (UID: \"c064c108-1022-402c-8c87-761962ef7747\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:58.704510 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.704482 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c064c108-1022-402c-8c87-761962ef7747-cert\") pod \"llmisvc-controller-manager-6466c554d8-s9485\" (UID: \"c064c108-1022-402c-8c87-761962ef7747\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:58.755157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.755116 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgbkq\" (UniqueName: \"kubernetes.io/projected/c064c108-1022-402c-8c87-761962ef7747-kube-api-access-fgbkq\") pod \"llmisvc-controller-manager-6466c554d8-s9485\" (UID: \"c064c108-1022-402c-8c87-761962ef7747\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:58.848784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:58.848729 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:09:59.222797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:09:59.222765 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s9485"] Oct 31 16:09:59.230386 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:09:59.230332 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc064c108_1022_402c_8c87_761962ef7747.slice/crio-55b862087c3a169c42208c21737209269d23931549bdcfd97853f1c28099a98e WatchSource:0}: Error finding container 55b862087c3a169c42208c21737209269d23931549bdcfd97853f1c28099a98e: Status 404 returned error can't find the container with id 55b862087c3a169c42208c21737209269d23931549bdcfd97853f1c28099a98e Oct 31 16:10:00.104241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:00.104204 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" event={"ID":"c064c108-1022-402c-8c87-761962ef7747","Type":"ContainerStarted","Data":"7de554d74197789b1909a714dd34a1791ee85a24b2c9fc9bb564f4c17e859ac3"} Oct 31 16:10:00.104687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:00.104293 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" event={"ID":"c064c108-1022-402c-8c87-761962ef7747","Type":"ContainerStarted","Data":"55b862087c3a169c42208c21737209269d23931549bdcfd97853f1c28099a98e"} Oct 31 16:10:00.104687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:00.104347 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:10:00.210317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:00.210267 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" podStartSLOduration=1.414459667 podStartE2EDuration="2.21025351s" totalImagesPullingTime="795.793843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:10:00.208533761 +0000 UTC m=+447.243828565" watchObservedRunningTime="2026-10-31 16:10:00.21025351 +0000 UTC m=+447.245548318" Oct 31 16:10:03.879590 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:10:03.879557 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:10:25.741836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:25.741801 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:31.111073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.111039 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s9485" Oct 31 16:10:31.288174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.288130 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5"] Oct 31 16:10:31.288441 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.288390 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" podUID="316dd107-ce44-4a87-8627-70c9620b349b" containerName="manager" containerID="cri-o://1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b" gracePeriod=30 Oct 31 16:10:31.532625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.532602 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:10:31.676969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.676893 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/316dd107-ce44-4a87-8627-70c9620b349b-cert\" (UniqueName: \"kubernetes.io/secret/316dd107-ce44-4a87-8627-70c9620b349b-cert\") pod \"316dd107-ce44-4a87-8627-70c9620b349b\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " Oct 31 16:10:31.676969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.676957 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316dd107-ce44-4a87-8627-70c9620b349b-kube-api-access-8zbm9\" (UniqueName: \"kubernetes.io/projected/316dd107-ce44-4a87-8627-70c9620b349b-kube-api-access-8zbm9\") pod \"316dd107-ce44-4a87-8627-70c9620b349b\" (UID: \"316dd107-ce44-4a87-8627-70c9620b349b\") " Oct 31 16:10:31.679024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.678996 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316dd107-ce44-4a87-8627-70c9620b349b-cert" pod "316dd107-ce44-4a87-8627-70c9620b349b" (UID: "316dd107-ce44-4a87-8627-70c9620b349b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:10:31.679156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.679032 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316dd107-ce44-4a87-8627-70c9620b349b-kube-api-access-8zbm9" pod "316dd107-ce44-4a87-8627-70c9620b349b" (UID: "316dd107-ce44-4a87-8627-70c9620b349b"). InnerVolumeSpecName "kube-api-access-8zbm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:10:31.777722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.777698 4880 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316dd107-ce44-4a87-8627-70c9620b349b-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:10:31.777722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:31.777723 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbm9\" (UniqueName: \"kubernetes.io/projected/316dd107-ce44-4a87-8627-70c9620b349b-kube-api-access-8zbm9\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:10:32.091193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.091098 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:32.254421 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.254388 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:32.271316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.271289 4880 generic.go:356] "Generic (PLEG): container finished" podID="316dd107-ce44-4a87-8627-70c9620b349b" containerID="1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b" exitCode=0 Oct 31 16:10:32.271446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.271351 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" event={"ID":"316dd107-ce44-4a87-8627-70c9620b349b","Type":"ContainerDied","Data":"1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b"} Oct 31 16:10:32.271446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.271360 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" Oct 31 16:10:32.271446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.271390 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5" event={"ID":"316dd107-ce44-4a87-8627-70c9620b349b","Type":"ContainerDied","Data":"88fd9768c81ccb93f0e06687f6b809a41eb8deea8434a9efc724566ee7c095bd"} Oct 31 16:10:32.271446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.271411 4880 scope.go:122] "RemoveContainer" containerID="1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b" Oct 31 16:10:32.281120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.281103 4880 scope.go:122] "RemoveContainer" containerID="1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b" Oct 31 16:10:32.281498 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:10:32.281473 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b\": container with ID starting with 1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b not found: ID does not exist" containerID="1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b" Oct 31 16:10:32.281570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.281511 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b"} err="failed to get container status \"1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b\": rpc error: code = NotFound desc = could not find container \"1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b\": container with ID starting with 1569e0db1b17283e665f8d9c297897b6d380c2092ffd018c981bdf1da884913b not found: ID does not exist" Oct 31 16:10:32.446234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.446200 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5"] Oct 31 16:10:32.499888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:32.499856 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5cjz5"] Oct 31 16:10:33.088989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:33.088958 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316dd107-ce44-4a87-8627-70c9620b349b" path="/var/lib/kubelet/pods/316dd107-ce44-4a87-8627-70c9620b349b/volumes" Oct 31 16:10:33.880711 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:10:33.880675 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:10:48.260885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:48.260849 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:48.359887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:48.359847 4880 generic.go:356] "Generic (PLEG): container finished" podID="cb9fb5ff-4889-4cf7-aa3a-37a80b636932" containerID="08e44b311dcdc5eb1a5c9b5bc0dac34f23beb2ea8be0527c6b1ad40156282fad" exitCode=1 Oct 31 16:10:48.360067 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:48.359922 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" event={"ID":"cb9fb5ff-4889-4cf7-aa3a-37a80b636932","Type":"ContainerDied","Data":"08e44b311dcdc5eb1a5c9b5bc0dac34f23beb2ea8be0527c6b1ad40156282fad"} Oct 31 16:10:48.360327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:48.360313 4880 scope.go:122] "RemoveContainer" containerID="08e44b311dcdc5eb1a5c9b5bc0dac34f23beb2ea8be0527c6b1ad40156282fad" Oct 31 16:10:48.463695 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:48.463664 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:49.182674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:49.182646 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:10:49.366817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:49.366781 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" event={"ID":"cb9fb5ff-4889-4cf7-aa3a-37a80b636932","Type":"ContainerStarted","Data":"8e16b23e974ecaf2e55bb482575dc9e541ba4736968bed5d2f59423131a5737b"} Oct 31 16:10:49.367251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:49.366870 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:10:50.788153 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:50.788115 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:54.624654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:54.624603 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:10:58.691452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:10:58.691418 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:00.374907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:00.374874 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8qmlh" Oct 31 16:11:01.637518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:01.637479 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:03.880675 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:11:03.880634 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:11:04.380196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:04.380129 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:07.131059 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:07.131024 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:10.352040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:10.352002 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:11.122850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:11.122809 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal" podUID="10be822f2f376ae4d26eb7bb3d2aeb64" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:11:13.501680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:13.501642 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:26.084908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:26.084873 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:33.879405 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:11:33.879370 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:11:40.458208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.458167 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6796fd4df4-27p22"] Oct 31 16:11:40.458772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.458748 4880 cpu_manager.go:414] "containerMap: removing container" podUID="316dd107-ce44-4a87-8627-70c9620b349b" containerName="manager" Oct 31 16:11:40.458825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.458776 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316dd107-ce44-4a87-8627-70c9620b349b" containerName="manager" Oct 31 16:11:40.458973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.458959 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="316dd107-ce44-4a87-8627-70c9620b349b" containerName="manager" Oct 31 16:11:40.462018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.461994 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.513275 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.513248 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6796fd4df4-27p22"] Oct 31 16:11:40.556896 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.556866 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hxc6d\"" Oct 31 16:11:40.557074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.557024 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 16:11:40.571119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.571082 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:40.571333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.571316 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61704b91-19e2-4915-9f4f-2a1dbf0307c6-tls-cert\") pod \"authorino-6796fd4df4-27p22\" (UID: \"61704b91-19e2-4915-9f4f-2a1dbf0307c6\") " pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.571378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.571352 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxm9\" (UniqueName: \"kubernetes.io/projected/61704b91-19e2-4915-9f4f-2a1dbf0307c6-kube-api-access-qbxm9\") pod \"authorino-6796fd4df4-27p22\" (UID: \"61704b91-19e2-4915-9f4f-2a1dbf0307c6\") " pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.672157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.672104 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61704b91-19e2-4915-9f4f-2a1dbf0307c6-tls-cert\") pod \"authorino-6796fd4df4-27p22\" (UID: \"61704b91-19e2-4915-9f4f-2a1dbf0307c6\") " pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.672342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.672169 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxm9\" (UniqueName: \"kubernetes.io/projected/61704b91-19e2-4915-9f4f-2a1dbf0307c6-kube-api-access-qbxm9\") pod \"authorino-6796fd4df4-27p22\" (UID: \"61704b91-19e2-4915-9f4f-2a1dbf0307c6\") " pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.674714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.674682 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/61704b91-19e2-4915-9f4f-2a1dbf0307c6-tls-cert\") pod \"authorino-6796fd4df4-27p22\" (UID: \"61704b91-19e2-4915-9f4f-2a1dbf0307c6\") " pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.724752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.724682 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxm9\" (UniqueName: \"kubernetes.io/projected/61704b91-19e2-4915-9f4f-2a1dbf0307c6-kube-api-access-qbxm9\") pod \"authorino-6796fd4df4-27p22\" (UID: \"61704b91-19e2-4915-9f4f-2a1dbf0307c6\") " pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.773278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.773244 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6796fd4df4-27p22" Oct 31 16:11:40.813292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:40.813205 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:41.136024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:41.135999 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6796fd4df4-27p22"] Oct 31 16:11:41.137540 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:11:41.137510 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61704b91_19e2_4915_9f4f_2a1dbf0307c6.slice/crio-1ced6556421ff811c47c79adc6f389ccee98f86c1c5573fe817dabca1c6aced0 WatchSource:0}: Error finding container 1ced6556421ff811c47c79adc6f389ccee98f86c1c5573fe817dabca1c6aced0: Status 404 returned error can't find the container with id 1ced6556421ff811c47c79adc6f389ccee98f86c1c5573fe817dabca1c6aced0 Oct 31 16:11:41.664553 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:41.664515 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6796fd4df4-27p22" event={"ID":"61704b91-19e2-4915-9f4f-2a1dbf0307c6","Type":"ContainerStarted","Data":"1ced6556421ff811c47c79adc6f389ccee98f86c1c5573fe817dabca1c6aced0"} Oct 31 16:11:43.676964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:43.676925 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6796fd4df4-27p22" event={"ID":"61704b91-19e2-4915-9f4f-2a1dbf0307c6","Type":"ContainerStarted","Data":"c3cc7ca5ba9472eb83da796873b1fe962829b65c73b501ffb933b138e99d621a"} Oct 31 16:11:43.781532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:43.781476 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6796fd4df4-27p22" podStartSLOduration=1.9058112189999998 podStartE2EDuration="3.781456991s" totalImagesPullingTime="1.875645772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:43.780775541 +0000 UTC m=+550.816070346" watchObservedRunningTime="2026-10-31 16:11:43.781456991 +0000 UTC m=+550.816751799" Oct 31 16:11:50.264853 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:50.264813 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:11:50.427771 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:11:50.427737 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:03.881055 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:12:03.881008 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:12:16.729021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:16.728935 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:17.211243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:17.211196 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:18.796114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:18.796081 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:18.939628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:18.939588 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:20.439316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:20.439282 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:20.683255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:20.683217 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:26.418776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:26.418736 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:33.143626 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:33.143595 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:12:33.144131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:33.144109 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:12:33.151295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:33.151272 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:12:33.151605 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:33.151590 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:12:33.164367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:33.164341 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:12:33.164643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:33.164623 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:12:33.879844 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:12:33.879803 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:12:44.770543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:44.770501 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:44.918783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:44.918746 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:45.362818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:45.362784 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:45.538395 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:45.538354 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:45.702605 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:45.702574 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:45.859898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:45.859865 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:46.011332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:46.011249 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:46.158759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:46.158720 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:46.308627 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:46.308536 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:46.567446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:46.567362 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:46.713156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:46.713099 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:46.947458 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:46.947421 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:47.090774 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:47.090742 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:47.233801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:47.233716 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:47.373752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:47.373715 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:47.519622 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:47.519539 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:48.960382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:48.960333 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:49.153923 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:49.153887 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:49.715888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:49.715853 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:49.922700 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:49.922649 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:50.680392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:50.680342 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:50.876801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:50.876766 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:51.034472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:51.034384 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:51.318786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:51.318702 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:51.545509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:51.545477 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:51.781399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:51.781364 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:52.158896 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:52.158857 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:52.337774 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:52.337734 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:52.491578 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:52.491490 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:53.877020 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:53.876986 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:54.094722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:54.094677 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:54.243385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:54.243350 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:54.408398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:54.408363 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:54.569614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:54.569521 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:54.726825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:54.726783 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:54.900449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:54.900413 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:56.299903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.299866 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5"] Oct 31 16:12:56.304095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.304073 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.352229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.352201 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-a1685e-serving-cert\"" Oct 31 16:12:56.367559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.367533 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4csp\" (UniqueName: \"kubernetes.io/projected/c36e1157-73f5-4539-8e47-de02d4da526a-kube-api-access-l4csp\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.367687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.367580 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36e1157-73f5-4539-8e47-de02d4da526a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.367735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.367718 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.367769 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.367743 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-metrics-certs\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.402418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.402383 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5"] Oct 31 16:12:56.468928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.468894 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.468928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.468933 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-metrics-certs\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.469171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.468958 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4csp\" (UniqueName: \"kubernetes.io/projected/c36e1157-73f5-4539-8e47-de02d4da526a-kube-api-access-l4csp\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.469171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.469000 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36e1157-73f5-4539-8e47-de02d4da526a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.469171 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:12:56.469064 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-serving-cert: secret "maas-api-e2e-worker-w6-a1685e-serving-cert" not found Oct 31 16:12:56.469302 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:12:56.469173 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls podName:c36e1157-73f5-4539-8e47-de02d4da526a nodeName:}" failed. No retries permitted until 2026-10-31 16:12:56.969152031 +0000 UTC m=+624.004446815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls") pod "maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" (UID: "c36e1157-73f5-4539-8e47-de02d4da526a") : secret "maas-api-e2e-worker-w6-a1685e-serving-cert" not found Oct 31 16:12:56.469713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.469685 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36e1157-73f5-4539-8e47-de02d4da526a-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.471459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.471432 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-metrics-certs\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.512223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.512192 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:56.523753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.523723 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4csp\" (UniqueName: \"kubernetes.io/projected/c36e1157-73f5-4539-8e47-de02d4da526a-kube-api-access-l4csp\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.667691 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.667656 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:56.820203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.820171 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:56.974010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.973920 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:56.976548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:56.976516 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:57.217115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:57.217084 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:57.371783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:57.371746 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:57.550783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:57.550695 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:57.587551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:57.587518 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5"] Oct 31 16:12:57.593641 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:12:57.593613 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36e1157_73f5_4539_8e47_de02d4da526a.slice/crio-a3c563d1b432673abc1c6bddbfc8ed151b1f90e0375be07f1d500952833e94d3 WatchSource:0}: Error finding container a3c563d1b432673abc1c6bddbfc8ed151b1f90e0375be07f1d500952833e94d3: Status 404 returned error can't find the container with id a3c563d1b432673abc1c6bddbfc8ed151b1f90e0375be07f1d500952833e94d3 Oct 31 16:12:57.594982 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:57.594962 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:12:57.709318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:57.709278 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:58.100020 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:58.099985 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" event={"ID":"c36e1157-73f5-4539-8e47-de02d4da526a","Type":"ContainerStarted","Data":"a3c563d1b432673abc1c6bddbfc8ed151b1f90e0375be07f1d500952833e94d3"} Oct 31 16:12:58.899261 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:58.899223 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:59.054104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:59.054012 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:12:59.107491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:59.107459 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" event={"ID":"c36e1157-73f5-4539-8e47-de02d4da526a","Type":"ContainerStarted","Data":"5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a"} Oct 31 16:12:59.107660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:59.107535 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:12:59.215856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:12:59.215803 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" podStartSLOduration=2.102615763 podStartE2EDuration="3.215789601s" totalImagesPullingTime="1.113173838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:59.215670801 +0000 UTC m=+626.250965609" watchObservedRunningTime="2026-10-31 16:12:59.215789601 +0000 UTC m=+626.251084406" Oct 31 16:13:01.267191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:01.267151 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:01.426744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:01.426710 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:02.024090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:02.024056 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:02.238748 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:02.238713 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:02.386337 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:02.386301 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:03.880501 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:03.880467 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:13:05.122497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:05.122465 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:13:05.791356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:05.791318 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:05.949714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:05.949678 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:07.297433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.297395 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp"] Oct 31 16:13:07.301184 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.301152 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.350351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.350322 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-ef6d93-serving-cert\"" Oct 31 16:13:07.373699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.373666 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a124856-37c9-4483-99c6-fa11207980cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.373881 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.373713 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc5hx\" (UniqueName: \"kubernetes.io/projected/4a124856-37c9-4483-99c6-fa11207980cf-kube-api-access-zc5hx\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.373951 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.373894 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.373951 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.373932 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-metrics-certs\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.398891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.398853 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp"] Oct 31 16:13:07.474937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.474907 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.475109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.474942 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-metrics-certs\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.475109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.474988 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a124856-37c9-4483-99c6-fa11207980cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.475329 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:07.475105 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-serving-cert: secret "maas-api-e2e-worker-w5-ef6d93-serving-cert" not found Oct 31 16:13:07.475329 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:07.475213 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls podName:4a124856-37c9-4483-99c6-fa11207980cf nodeName:}" failed. No retries permitted until 2026-10-31 16:13:07.975189302 +0000 UTC m=+635.010484087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls") pod "maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" (UID: "4a124856-37c9-4483-99c6-fa11207980cf") : secret "maas-api-e2e-worker-w5-ef6d93-serving-cert" not found Oct 31 16:13:07.475329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.475114 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc5hx\" (UniqueName: \"kubernetes.io/projected/4a124856-37c9-4483-99c6-fa11207980cf-kube-api-access-zc5hx\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.476698 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.476677 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a124856-37c9-4483-99c6-fa11207980cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.478784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.478760 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-metrics-certs\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.528293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.528268 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc5hx\" (UniqueName: \"kubernetes.io/projected/4a124856-37c9-4483-99c6-fa11207980cf-kube-api-access-zc5hx\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.549650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.549590 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:07.740199 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.740153 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:07.881973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.881887 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:07.979391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.979358 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:07.981727 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:07.981705 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls\") pod \"maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:08.213949 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:08.213909 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:08.584600 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:08.584513 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp"] Oct 31 16:13:08.590269 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:13:08.590241 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a124856_37c9_4483_99c6_fa11207980cf.slice/crio-48598615bc5da5fff76d96f46805fb819c01450cefc4686411dfd390089bb320 WatchSource:0}: Error finding container 48598615bc5da5fff76d96f46805fb819c01450cefc4686411dfd390089bb320: Status 404 returned error can't find the container with id 48598615bc5da5fff76d96f46805fb819c01450cefc4686411dfd390089bb320 Oct 31 16:13:09.168028 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:09.167991 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" event={"ID":"4a124856-37c9-4483-99c6-fa11207980cf","Type":"ContainerStarted","Data":"48598615bc5da5fff76d96f46805fb819c01450cefc4686411dfd390089bb320"} Oct 31 16:13:10.175440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:10.175405 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" event={"ID":"4a124856-37c9-4483-99c6-fa11207980cf","Type":"ContainerStarted","Data":"416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d"} Oct 31 16:13:10.175823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:10.175495 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:10.284225 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:10.284169 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" podStartSLOduration=2.495187389 podStartE2EDuration="3.284153128s" totalImagesPullingTime="788.965739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:10.282180392 +0000 UTC m=+637.317475200" watchObservedRunningTime="2026-10-31 16:13:10.284153128 +0000 UTC m=+637.319447933" Oct 31 16:13:10.380695 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:10.380658 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:10.537274 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:10.537182 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:12.683267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:12.683226 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:12.872298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:12.872265 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:12.890769 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:12.890731 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b"] Oct 31 16:13:12.894843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:12.894824 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:12.992756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:12.992678 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ac1273-serving-cert\"" Oct 31 16:13:12.995103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:12.995065 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b"] Oct 31 16:13:13.026445 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.026410 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.026584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.026458 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2tk\" (UniqueName: \"kubernetes.io/projected/0a301a2c-fca6-4540-8362-88bd33ff735e-kube-api-access-5w2tk\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.026584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.026478 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.026659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.026622 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a301a2c-fca6-4540-8362-88bd33ff735e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.127933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.127898 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.128119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.127943 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2tk\" (UniqueName: \"kubernetes.io/projected/0a301a2c-fca6-4540-8362-88bd33ff735e-kube-api-access-5w2tk\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.128119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.128081 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.128291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.128198 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a301a2c-fca6-4540-8362-88bd33ff735e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.128291 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:13.128213 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-serving-cert: secret "maas-api-e2e-worker-w0-ac1273-serving-cert" not found Oct 31 16:13:13.128291 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:13.128278 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls podName:0a301a2c-fca6-4540-8362-88bd33ff735e nodeName:}" failed. No retries permitted until 2026-10-31 16:13:13.628256408 +0000 UTC m=+640.663551205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls") pod "maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" (UID: "0a301a2c-fca6-4540-8362-88bd33ff735e") : secret "maas-api-e2e-worker-w0-ac1273-serving-cert" not found Oct 31 16:13:13.128937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.128911 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a301a2c-fca6-4540-8362-88bd33ff735e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.130714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.130692 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-metrics-certs\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.175775 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.175717 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:13.181095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.181068 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2tk\" (UniqueName: \"kubernetes.io/projected/0a301a2c-fca6-4540-8362-88bd33ff735e-kube-api-access-5w2tk\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.336487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.336397 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:13.476474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.476442 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:13.633351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.633271 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.635776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.635752 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:13.807317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:13.807271 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:14.200895 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:14.200858 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b"] Oct 31 16:13:14.206068 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:13:14.206030 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a301a2c_fca6_4540_8362_88bd33ff735e.slice/crio-89bcdca5eafcc889b029034b7e5db9d545914314fdfe6d546173b51f781ae8fa WatchSource:0}: Error finding container 89bcdca5eafcc889b029034b7e5db9d545914314fdfe6d546173b51f781ae8fa: Status 404 returned error can't find the container with id 89bcdca5eafcc889b029034b7e5db9d545914314fdfe6d546173b51f781ae8fa Oct 31 16:13:15.063923 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:15.063880 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:15.205594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:15.205561 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" event={"ID":"0a301a2c-fca6-4540-8362-88bd33ff735e","Type":"ContainerStarted","Data":"5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4"} Oct 31 16:13:15.205767 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:15.205611 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" event={"ID":"0a301a2c-fca6-4540-8362-88bd33ff735e","Type":"ContainerStarted","Data":"89bcdca5eafcc889b029034b7e5db9d545914314fdfe6d546173b51f781ae8fa"} Oct 31 16:13:15.205767 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:15.205689 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:15.208090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:15.208054 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:15.313802 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:15.313745 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" podStartSLOduration=2.428004818 podStartE2EDuration="3.313732608s" totalImagesPullingTime="885.72779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:15.311073001 +0000 UTC m=+642.346367825" watchObservedRunningTime="2026-10-31 16:13:15.313732608 +0000 UTC m=+642.349027412" Oct 31 16:13:16.186461 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:16.186431 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:13:18.754456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:18.754414 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:21.217096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:21.217070 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:13:23.809962 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:23.809926 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:23.979116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:23.979076 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:24.115924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:24.115808 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:26.747816 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:26.747782 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:28.593926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.593895 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws"] Oct 31 16:13:28.598875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.598849 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.676731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.676690 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.676731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.676732 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-metrics-certs\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.676977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.676844 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wpp\" (UniqueName: \"kubernetes.io/projected/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-kube-api-access-p8wpp\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.676977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.676895 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.693229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.693193 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e7f13a-serving-cert\"" Oct 31 16:13:28.696022 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.695994 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws"] Oct 31 16:13:28.777684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.777650 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.777684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.777688 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-metrics-certs\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.777940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.777743 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wpp\" (UniqueName: \"kubernetes.io/projected/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-kube-api-access-p8wpp\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.777940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.777785 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.777940 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:28.777898 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-serving-cert: secret "maas-api-e2e-worker-w2-e7f13a-serving-cert" not found Oct 31 16:13:28.778085 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:28.777968 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls podName:3c5ec607-72b9-44e6-ad3f-0dd99140ab00 nodeName:}" failed. No retries permitted until 2026-10-31 16:13:29.277948423 +0000 UTC m=+656.313243220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls") pod "maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" (UID: "3c5ec607-72b9-44e6-ad3f-0dd99140ab00") : secret "maas-api-e2e-worker-w2-e7f13a-serving-cert" not found Oct 31 16:13:28.778389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.778367 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.780337 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.780314 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-metrics-certs\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:28.817525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.817490 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:28.832089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:28.832054 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wpp\" (UniqueName: \"kubernetes.io/projected/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-kube-api-access-p8wpp\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:29.012105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:29.012069 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:29.156152 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:29.156093 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:29.285944 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:29.283410 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:29.291098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:29.291068 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:29.512282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:29.512239 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:29.880863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:29.880769 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws"] Oct 31 16:13:29.887828 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:13:29.887799 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5ec607_72b9_44e6_ad3f_0dd99140ab00.slice/crio-18ceed79f4f4063409f00eb324752bd555be38fd0e0b06ac68ae47f1d62badce WatchSource:0}: Error finding container 18ceed79f4f4063409f00eb324752bd555be38fd0e0b06ac68ae47f1d62badce: Status 404 returned error can't find the container with id 18ceed79f4f4063409f00eb324752bd555be38fd0e0b06ac68ae47f1d62badce Oct 31 16:13:30.300415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:30.300379 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" event={"ID":"3c5ec607-72b9-44e6-ad3f-0dd99140ab00","Type":"ContainerStarted","Data":"18ceed79f4f4063409f00eb324752bd555be38fd0e0b06ac68ae47f1d62badce"} Oct 31 16:13:31.309164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.309087 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" event={"ID":"3c5ec607-72b9-44e6-ad3f-0dd99140ab00","Type":"ContainerStarted","Data":"3a20984cc80b4d14cd77b87f9e72cef73a3fbdbe9fa6dbfc964136f4ee2c41f5"} Oct 31 16:13:31.309670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.309362 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:31.415328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.415280 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" podStartSLOduration=2.5362612650000003 podStartE2EDuration="3.415265936s" totalImagesPullingTime="879.004671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:31.41457313 +0000 UTC m=+658.449867935" watchObservedRunningTime="2026-10-31 16:13:31.415265936 +0000 UTC m=+658.450560780" Oct 31 16:13:31.657255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.657214 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4"] Oct 31 16:13:31.661281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.661256 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.703815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.703779 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.703971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.703835 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5072770-07b2-4574-800f-b34f39fbba10-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.704033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.703966 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.704033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.704009 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbsz\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-kube-api-access-lwbsz\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.704121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.704107 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.704272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.704207 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.704341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.704323 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.704378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.704359 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.704422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.704391 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5072770-07b2-4574-800f-b34f39fbba10-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.709369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.709351 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ac1273-openshift-default-dockercfg-zfcwg\"" Oct 31 16:13:31.735266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.735234 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4"] Oct 31 16:13:31.805315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805283 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805323 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbsz\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-kube-api-access-lwbsz\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805432 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805456 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805478 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805497 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805520 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5072770-07b2-4574-800f-b34f39fbba10-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805573 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805604 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5072770-07b2-4574-800f-b34f39fbba10-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805869 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805766 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805869 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805844 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805954 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.805979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.805962 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.806365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.806340 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5072770-07b2-4574-800f-b34f39fbba10-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.807999 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.807980 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.808124 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.808105 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5072770-07b2-4574-800f-b34f39fbba10-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.859435 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.859395 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.859574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.859438 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbsz\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-kube-api-access-lwbsz\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:31.974746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:31.974656 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:32.413295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:32.413253 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4"] Oct 31 16:13:32.419055 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:13:32.419025 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5072770_07b2_4574_800f_b34f39fbba10.slice/crio-d8f62f80fbbca54e1380c7176428012bda7d9ca10b1601b57ed6d8d226423f27 WatchSource:0}: Error finding container d8f62f80fbbca54e1380c7176428012bda7d9ca10b1601b57ed6d8d226423f27: Status 404 returned error can't find the container with id d8f62f80fbbca54e1380c7176428012bda7d9ca10b1601b57ed6d8d226423f27 Oct 31 16:13:32.422167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:32.422097 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:13:32.422288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:32.422233 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:13:32.422353 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:32.422287 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:13:33.356613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:33.356578 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" event={"ID":"a5072770-07b2-4574-800f-b34f39fbba10","Type":"ContainerStarted","Data":"397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb"} Oct 31 16:13:33.356837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:33.356635 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" event={"ID":"a5072770-07b2-4574-800f-b34f39fbba10","Type":"ContainerStarted","Data":"d8f62f80fbbca54e1380c7176428012bda7d9ca10b1601b57ed6d8d226423f27"} Oct 31 16:13:33.557450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:33.557388 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podStartSLOduration=2.5573697380000002 podStartE2EDuration="2.557369738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:33.510537479 +0000 UTC m=+660.545832283" watchObservedRunningTime="2026-10-31 16:13:33.557369738 +0000 UTC m=+660.592664543" Oct 31 16:13:33.846330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:33.846287 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:33.882425 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:13:33.882388 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:13:33.975684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:33.975650 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:33.977586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:33.977552 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 16:13:33.977721 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:33.977609 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 16:13:34.006827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:34.006793 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:34.380150 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:34.380110 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:34.976001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:34.975960 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 16:13:34.976427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:34.976016 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 16:13:35.975889 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:35.975852 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 16:13:35.976101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:35.975917 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 16:13:36.976055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:36.976021 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 16:13:36.976241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:36.976081 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 16:13:37.350352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:37.350255 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:13:37.975562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:37.975518 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 16:13:37.975767 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:37.975580 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 16:13:38.979894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:38.979861 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:38.980371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:38.980350 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:38.981107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:38.981089 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:13:39.061275 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:39.061235 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:43.971980 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:43.971894 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:45.372821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:45.372781 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:45.989452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:45.989417 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:46.142179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:46.142114 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:46.306289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:46.306192 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:49.313851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:49.313816 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:49.979804 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:49.979770 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg"] Oct 31 16:13:49.983995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:49.983973 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.079517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.079484 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"default-dockercfg-7hqvj\"" Oct 31 16:13:50.079517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.079484 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"simulator-w6-a1685e-kserve-self-signed-certs\"" Oct 31 16:13:50.081968 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.081943 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg"] Oct 31 16:13:50.084008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.083990 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"openshift-service-ca.crt\"" Oct 31 16:13:50.084120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.084060 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"kube-root-ca.crt\"" Oct 31 16:13:50.094060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.094039 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f065a714-871a-44eb-be2d-1f81d40e3a2b-tls-certs\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.094172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.094097 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-dshm\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.094242 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.094188 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-model-cache\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.094291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.094246 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmln\" (UniqueName: \"kubernetes.io/projected/f065a714-871a-44eb-be2d-1f81d40e3a2b-kube-api-access-hpmln\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.094329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.094296 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-tmp-dir\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.094407 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.094385 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-home\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.097635 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.097610 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:50.195543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.195513 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmln\" (UniqueName: \"kubernetes.io/projected/f065a714-871a-44eb-be2d-1f81d40e3a2b-kube-api-access-hpmln\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.195751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.195560 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-tmp-dir\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.195751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.195578 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-home\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.195751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.195710 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f065a714-871a-44eb-be2d-1f81d40e3a2b-tls-certs\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.195924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.195791 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-dshm\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.195924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.195826 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-model-cache\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.196030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.195974 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-home\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.196086 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.196065 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-tmp-dir\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.196206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.196187 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-model-cache\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.197961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.197941 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-dshm\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.198131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.198109 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f065a714-871a-44eb-be2d-1f81d40e3a2b-tls-certs\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.247991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.247900 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmln\" (UniqueName: \"kubernetes.io/projected/f065a714-871a-44eb-be2d-1f81d40e3a2b-kube-api-access-hpmln\") pod \"simulator-w6-a1685e-kserve-8856947c5-4xrxg\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.296773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.296734 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:50.665871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:50.665838 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg"] Oct 31 16:13:50.671783 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:13:50.671755 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf065a714_871a_44eb_be2d_1f81d40e3a2b.slice/crio-f550e166a62f8c0c86e1500c4bbae5bb9f8ba818493fa9910f90ef364e2789a8 WatchSource:0}: Error finding container f550e166a62f8c0c86e1500c4bbae5bb9f8ba818493fa9910f90ef364e2789a8: Status 404 returned error can't find the container with id f550e166a62f8c0c86e1500c4bbae5bb9f8ba818493fa9910f90ef364e2789a8 Oct 31 16:13:51.329935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:51.329906 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:51.469829 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:51.469784 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" event={"ID":"f065a714-871a-44eb-be2d-1f81d40e3a2b","Type":"ContainerStarted","Data":"f550e166a62f8c0c86e1500c4bbae5bb9f8ba818493fa9910f90ef364e2789a8"} Oct 31 16:13:51.949703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:51.949657 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:52.105218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:52.105178 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:52.477039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:52.476997 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" event={"ID":"f065a714-871a-44eb-be2d-1f81d40e3a2b","Type":"ContainerStarted","Data":"b4af165fd56b353523afe4877f467109d41f3fec7e99f32fa14df453cb562fcc"} Oct 31 16:13:52.587395 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:52.587295 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podStartSLOduration=2.022337408 podStartE2EDuration="3.587280381s" totalImagesPullingTime="1.564942973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:52.585247021 +0000 UTC m=+679.620541827" watchObservedRunningTime="2026-10-31 16:13:52.587280381 +0000 UTC m=+679.622575207" Oct 31 16:13:53.297483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:53.297451 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:13:54.856090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:54.856049 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:57.766715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:57.766667 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:58.367178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:58.367123 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:13:58.564207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:13:58.564170 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:00.170047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:00.170008 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:00.297218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:00.297185 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:14:00.309252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:00.309231 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:14:00.536280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:00.536202 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:14:01.704628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:01.704591 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:02.304935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:02.304904 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:02.471158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:02.471117 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:02.619209 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:02.619117 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:03.881299 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:14:03.881262 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:14:05.689556 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:05.689509 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:10.682876 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:10.682831 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:15.975027 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:15.974990 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:16.194411 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.194380 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:16.439375 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.439339 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp"] Oct 31 16:14:16.442911 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.442887 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.496764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.496732 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp"] Oct 31 16:14:16.537764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.537730 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"openshift-service-ca.crt\"" Oct 31 16:14:16.537995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.537937 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"kube-root-ca.crt\"" Oct 31 16:14:16.538567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.538549 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"simulator-w3-f7821a-kserve-self-signed-certs\"" Oct 31 16:14:16.538679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.538579 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"default-dockercfg-8rxfz\"" Oct 31 16:14:16.553011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.552980 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-dshm\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.553159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.553033 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tmp-dir\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.553226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.553202 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plpqr\" (UniqueName: \"kubernetes.io/projected/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-kube-api-access-plpqr\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.553263 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.553248 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tls-certs\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.553301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.553295 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-home\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.553337 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.553317 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-model-cache\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.555465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.555441 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:16.654163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654098 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tls-certs\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654161 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-home\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654197 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-model-cache\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654271 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-dshm\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654337 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tmp-dir\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654423 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpqr\" (UniqueName: \"kubernetes.io/projected/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-kube-api-access-plpqr\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654574 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-home\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654624 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-model-cache\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.654877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.654855 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tmp-dir\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.656683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.656660 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-dshm\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.656946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.656925 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tls-certs\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.708745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.708661 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plpqr\" (UniqueName: \"kubernetes.io/projected/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-kube-api-access-plpqr\") pod \"simulator-w3-f7821a-kserve-779686fb7-pfqwp\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.754465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.754427 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:16.782240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.782206 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:16.944475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:16.944442 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:17.103793 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:17.103692 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:17.132623 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:17.132589 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp"] Oct 31 16:14:17.138184 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:14:17.138156 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd4f876b_52f7_41af_bd1f_63ff7ccc42be.slice/crio-5dc634283eee97483f5eb1adbed074673a85ff7f684e4285c776342e2a134675 WatchSource:0}: Error finding container 5dc634283eee97483f5eb1adbed074673a85ff7f684e4285c776342e2a134675: Status 404 returned error can't find the container with id 5dc634283eee97483f5eb1adbed074673a85ff7f684e4285c776342e2a134675 Oct 31 16:14:17.625502 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:17.625464 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" event={"ID":"cd4f876b-52f7-41af-bd1f-63ff7ccc42be","Type":"ContainerStarted","Data":"f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac"} Oct 31 16:14:17.625703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:17.625517 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" event={"ID":"cd4f876b-52f7-41af-bd1f-63ff7ccc42be","Type":"ContainerStarted","Data":"5dc634283eee97483f5eb1adbed074673a85ff7f684e4285c776342e2a134675"} Oct 31 16:14:17.731172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:17.731095 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podStartSLOduration=1.731080066 podStartE2EDuration="1.731080066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:17.730480903 +0000 UTC m=+704.765775709" watchObservedRunningTime="2026-10-31 16:14:17.731080066 +0000 UTC m=+704.766374871" Oct 31 16:14:17.755548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:17.755512 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:20.575721 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.575685 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q"] Oct 31 16:14:20.580026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.580003 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.675167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.675100 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"openshift-service-ca.crt\"" Oct 31 16:14:20.675167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.675162 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"kube-root-ca.crt\"" Oct 31 16:14:20.675422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.675286 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"simulator-w5-ef6d93-kserve-self-signed-certs\"" Oct 31 16:14:20.675480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.675467 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6d93\"/\"default-dockercfg-hnnx7\"" Oct 31 16:14:20.678339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.678313 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q"] Oct 31 16:14:20.684435 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.684409 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:20.694550 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.694527 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-model-cache\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.694666 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.694564 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tmp-dir\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.694666 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.694588 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tls-certs\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.694752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.694723 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-dshm\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.694808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.694753 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wx5p\" (UniqueName: \"kubernetes.io/projected/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-kube-api-access-6wx5p\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.694808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.694783 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-home\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.795703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.795661 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-dshm\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.795892 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.795721 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx5p\" (UniqueName: \"kubernetes.io/projected/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-kube-api-access-6wx5p\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.795892 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.795755 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-home\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.795892 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.795865 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-model-cache\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.796074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.795912 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tmp-dir\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.796074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.795948 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tls-certs\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.796250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.796227 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-home\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.796317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.796289 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-model-cache\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.796369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.796357 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tmp-dir\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.798067 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.798041 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-dshm\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.798340 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.798323 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tls-certs\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.835718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.835642 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:20.848574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.848544 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wx5p\" (UniqueName: \"kubernetes.io/projected/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-kube-api-access-6wx5p\") pod \"simulator-w5-ef6d93-kserve-79df45d547-smx4q\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:20.895273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:20.895237 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:21.275362 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.275328 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q"] Oct 31 16:14:21.280944 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:14:21.280903 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40c4a6ca_af68_4383_bcb9_dd84893a0fc6.slice/crio-0c44681c632f85ba8f92463854689a8bd49b0a4273bf4875d089832fd28d216f WatchSource:0}: Error finding container 0c44681c632f85ba8f92463854689a8bd49b0a4273bf4875d089832fd28d216f: Status 404 returned error can't find the container with id 0c44681c632f85ba8f92463854689a8bd49b0a4273bf4875d089832fd28d216f Oct 31 16:14:21.284635 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.284598 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:21.444968 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.444925 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:21.594665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.594586 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:21.660833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.660802 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" event={"ID":"40c4a6ca-af68-4383-bcb9-dd84893a0fc6","Type":"ContainerStarted","Data":"1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883"} Oct 31 16:14:21.661030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.660846 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" event={"ID":"40c4a6ca-af68-4383-bcb9-dd84893a0fc6","Type":"ContainerStarted","Data":"0c44681c632f85ba8f92463854689a8bd49b0a4273bf4875d089832fd28d216f"} Oct 31 16:14:21.770559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.770501 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podStartSLOduration=1.770484467 podStartE2EDuration="1.770484467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:21.767330227 +0000 UTC m=+708.802625032" watchObservedRunningTime="2026-10-31 16:14:21.770484467 +0000 UTC m=+708.805779273" Oct 31 16:14:21.895805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:21.895771 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:22.809060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:22.809021 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:23.397147 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:23.397088 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:23.554370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:23.554340 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:23.707235 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:23.707150 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:26.571922 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:26.571889 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:26.754832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:26.754790 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:26.767237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:26.767204 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:27.071549 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:27.071513 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:27.707577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:27.707546 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:14:30.896442 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:30.896408 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:30.909133 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:30.909103 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:31.471641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:31.471604 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:31.708842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:31.708807 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:31.734314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:31.734241 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:14:33.881506 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:14:33.881470 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:14:36.360276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:36.360234 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:41.285870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:41.285835 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:44.389734 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.389691 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875"] Oct 31 16:14:44.395450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.395424 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.446028 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.445994 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875"] Oct 31 16:14:44.488687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.488644 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"simulator-w1-4fe3a4-kserve-self-signed-certs\"" Oct 31 16:14:44.488687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.488683 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"default-dockercfg-jbp9j\"" Oct 31 16:14:44.490315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.490297 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"openshift-service-ca.crt\"" Oct 31 16:14:44.490394 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.490382 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"kube-root-ca.crt\"" Oct 31 16:14:44.493800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.493777 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:44.526010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.525979 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswqh\" (UniqueName: \"kubernetes.io/projected/05265cb6-63be-4ce4-b005-673ece77fb83-kube-api-access-mswqh\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.526178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.526035 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-home\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.526178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.526080 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-model-cache\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.526283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.526233 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05265cb6-63be-4ce4-b005-673ece77fb83-tls-certs\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.526283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.526261 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-dshm\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.526364 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.526294 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-tmp-dir\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.627449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.627412 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05265cb6-63be-4ce4-b005-673ece77fb83-tls-certs\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.627449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.627452 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-dshm\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.627667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.627556 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-tmp-dir\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.627667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.627606 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswqh\" (UniqueName: \"kubernetes.io/projected/05265cb6-63be-4ce4-b005-673ece77fb83-kube-api-access-mswqh\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.627667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.627660 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-home\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.627798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.627682 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-model-cache\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.628075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.628049 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-tmp-dir\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.628075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.628069 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-model-cache\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.628250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.628097 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-home\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.629852 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.629832 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-dshm\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.630036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.630018 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05265cb6-63be-4ce4-b005-673ece77fb83-tls-certs\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.686302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.686228 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswqh\" (UniqueName: \"kubernetes.io/projected/05265cb6-63be-4ce4-b005-673ece77fb83-kube-api-access-mswqh\") pod \"simulator-w1-4fe3a4-kserve-544f5d68cc-wt875\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:44.707099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:44.707057 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:45.090332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:45.090228 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875"] Oct 31 16:14:45.093188 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:14:45.093157 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05265cb6_63be_4ce4_b005_673ece77fb83.slice/crio-9e08e258a14d4810ee6225a2e1cc2346aed28e16751678cbebe1057dcc0a56ed WatchSource:0}: Error finding container 9e08e258a14d4810ee6225a2e1cc2346aed28e16751678cbebe1057dcc0a56ed: Status 404 returned error can't find the container with id 9e08e258a14d4810ee6225a2e1cc2346aed28e16751678cbebe1057dcc0a56ed Oct 31 16:14:45.815621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:45.815579 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" event={"ID":"05265cb6-63be-4ce4-b005-673ece77fb83","Type":"ContainerStarted","Data":"d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508"} Oct 31 16:14:45.816035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:45.815639 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" event={"ID":"05265cb6-63be-4ce4-b005-673ece77fb83","Type":"ContainerStarted","Data":"9e08e258a14d4810ee6225a2e1cc2346aed28e16751678cbebe1057dcc0a56ed"} Oct 31 16:14:45.923196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:45.923126 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podStartSLOduration=1.923111609 podStartE2EDuration="1.923111609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:45.920633217 +0000 UTC m=+732.955928025" watchObservedRunningTime="2026-10-31 16:14:45.923111609 +0000 UTC m=+732.958406414" Oct 31 16:14:46.346700 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:46.346666 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:46.707678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:46.707643 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:51.247440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:51.247402 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:54.707358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:54.707324 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:54.720078 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:54.720057 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:54.887640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:54.887613 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:14:56.070327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:56.070290 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:14:57.746174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:14:57.746114 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:00.383103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.383071 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p"] Oct 31 16:15:00.387218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.387199 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.439910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.439877 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p"] Oct 31 16:15:00.481918 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.481887 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"default-dockercfg-mj79l\"" Oct 31 16:15:00.482091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.482041 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"kube-root-ca.crt\"" Oct 31 16:15:00.482091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.482059 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"openshift-service-ca.crt\"" Oct 31 16:15:00.482091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.482060 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7f13a\"/\"simulator-w2-e7f13a-kserve-self-signed-certs\"" Oct 31 16:15:00.498386 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.498344 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:00.500454 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.500436 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-home\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.500534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.500474 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-model-cache\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.500579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.500570 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tls-certs\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.500664 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.500649 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tmp-dir\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.500705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.500673 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grr5\" (UniqueName: \"kubernetes.io/projected/aa80a393-96cf-4c65-8d56-84d6ca0292b8-kube-api-access-8grr5\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.500705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.500698 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-dshm\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601243 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tls-certs\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601298 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tmp-dir\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601315 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grr5\" (UniqueName: \"kubernetes.io/projected/aa80a393-96cf-4c65-8d56-84d6ca0292b8-kube-api-access-8grr5\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601343 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-dshm\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601391 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-home\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601435 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-model-cache\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601768 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601742 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tmp-dir\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601831 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-home\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.601910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.601878 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-model-cache\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.603721 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.603695 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-dshm\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.604001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.603978 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tls-certs\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.654115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.654086 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grr5\" (UniqueName: \"kubernetes.io/projected/aa80a393-96cf-4c65-8d56-84d6ca0292b8-kube-api-access-8grr5\") pod \"simulator-w2-e7f13a-kserve-69dc6496c4-68f2p\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.700030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.699988 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:00.889276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:00.889242 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:01.078908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:01.078834 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p"] Oct 31 16:15:01.084946 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:15:01.084919 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa80a393_96cf_4c65_8d56_84d6ca0292b8.slice/crio-966b3ebe99fc2e0dbd97f69d9fa0509615d33b72a7532343c1fd7017b87dfcdf WatchSource:0}: Error finding container 966b3ebe99fc2e0dbd97f69d9fa0509615d33b72a7532343c1fd7017b87dfcdf: Status 404 returned error can't find the container with id 966b3ebe99fc2e0dbd97f69d9fa0509615d33b72a7532343c1fd7017b87dfcdf Oct 31 16:15:01.930664 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:01.930625 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" event={"ID":"aa80a393-96cf-4c65-8d56-84d6ca0292b8","Type":"ContainerStarted","Data":"1afb3413fe539817ebcd0564cbe38b94c3605517c62623ee2e9fb9d598d2b1f8"} Oct 31 16:15:01.931053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:01.930677 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" event={"ID":"aa80a393-96cf-4c65-8d56-84d6ca0292b8","Type":"ContainerStarted","Data":"966b3ebe99fc2e0dbd97f69d9fa0509615d33b72a7532343c1fd7017b87dfcdf"} Oct 31 16:15:02.037958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:02.037905 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podStartSLOduration=2.037890796 podStartE2EDuration="2.037890796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:02.036281082 +0000 UTC m=+749.071575888" watchObservedRunningTime="2026-10-31 16:15:02.037890796 +0000 UTC m=+749.073185602" Oct 31 16:15:02.701168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:02.701104 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:03.879968 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:15:03.879929 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:15:05.675440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:05.675407 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:06.197809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:06.197764 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:10.548289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:10.548248 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:10.700859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:10.700833 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:10.714118 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:10.714092 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:10.995270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:10.995243 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:15:15.342324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:15.342236 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:18.214559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.214519 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg"] Oct 31 16:15:18.218653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.218625 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.287415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.287372 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-model-cache\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.287581 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.287503 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tmp-dir\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.287581 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.287557 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-dshm\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.287663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.287584 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tls-certs\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.287718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.287698 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scjb\" (UniqueName: \"kubernetes.io/projected/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-kube-api-access-5scjb\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.287781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.287764 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-home\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.313826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.313794 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f7821a\"/\"premium-w3-f7821a-kserve-self-signed-certs\"" Oct 31 16:15:18.316703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.316682 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg"] Oct 31 16:15:18.320328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.320299 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:18.389321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389290 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scjb\" (UniqueName: \"kubernetes.io/projected/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-kube-api-access-5scjb\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389356 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-home\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389401 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-model-cache\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389448 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tmp-dir\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389486 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-dshm\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389595 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tls-certs\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389744 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-home\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389850 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tmp-dir\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.389937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.389913 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-model-cache\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.391553 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.391531 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-dshm\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.391883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.391867 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tls-certs\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.442249 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.442210 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scjb\" (UniqueName: \"kubernetes.io/projected/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-kube-api-access-5scjb\") pod \"premium-w3-f7821a-kserve-5844fd97bb-jk5lg\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.531457 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.531369 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:18.918246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:18.918213 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg"] Oct 31 16:15:18.924189 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:15:18.924163 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c36c9a_01b5_42eb_9ae9_23717f85bd8c.slice/crio-329a0411bcf49d82a3849b7366c174390873a416a1364e28362a93f26351d058 WatchSource:0}: Error finding container 329a0411bcf49d82a3849b7366c174390873a416a1364e28362a93f26351d058: Status 404 returned error can't find the container with id 329a0411bcf49d82a3849b7366c174390873a416a1364e28362a93f26351d058 Oct 31 16:15:19.035889 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:19.035857 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" event={"ID":"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c","Type":"ContainerStarted","Data":"cdf58fc26c6e63791932aff77282162091391429802ce6611e8825f06b4b8e3a"} Oct 31 16:15:19.036043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:19.035903 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" event={"ID":"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c","Type":"ContainerStarted","Data":"329a0411bcf49d82a3849b7366c174390873a416a1364e28362a93f26351d058"} Oct 31 16:15:19.141637 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:19.141575 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podStartSLOduration=1.14156204 podStartE2EDuration="1.14156204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:19.139927739 +0000 UTC m=+766.175222542" watchObservedRunningTime="2026-10-31 16:15:19.14156204 +0000 UTC m=+766.176856865" Oct 31 16:15:19.531593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:19.531563 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:20.247556 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:20.247525 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:25.147896 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:25.147859 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:27.443746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:27.443710 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:28.531962 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:28.531923 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:28.544460 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:28.544433 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:29.107112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:29.107084 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:15:30.042552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:30.042516 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:33.880839 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:15:33.880794 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:15:34.842019 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:34.841987 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:35.901070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:35.901035 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:36.854023 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:36.853991 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:36.994228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:36.994177 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:39.661124 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:39.661085 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:44.459966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:44.459932 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:49.356115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:49.356077 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:51.478924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:51.478875 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:51.620068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:51.620033 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:51.843977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:51.843890 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:51.987192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:51.987159 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:52.382594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:52.382561 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:54.133032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:54.132997 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:15:58.940746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:15:58.940706 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:03.846495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:03.846440 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:03.881433 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:16:03.881396 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:16:03.971555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:03.971522 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f"] Oct 31 16:16:03.975858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:03.975837 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.028077 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.028039 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f"] Oct 31 16:16:04.052080 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.052051 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-home\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.052287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.052109 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tls-certs\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.052287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.052211 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-dshm\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.052287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.052250 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gt4c\" (UniqueName: \"kubernetes.io/projected/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-kube-api-access-2gt4c\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.052416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.052307 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-model-cache\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.052416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.052358 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tmp-dir\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.069829 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.069793 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4fe3a4\"/\"premium-w1-4fe3a4-kserve-self-signed-certs\"" Oct 31 16:16:04.084957 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.084925 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:04.153548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.153510 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-home\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.155609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.155451 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-home\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.155758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.155731 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tls-certs\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.155833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.155771 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-dshm\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.155833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.155815 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gt4c\" (UniqueName: \"kubernetes.io/projected/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-kube-api-access-2gt4c\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.155942 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.155890 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-model-cache\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.155999 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.155973 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tmp-dir\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.156451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.156351 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tmp-dir\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.156569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.156523 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-model-cache\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.163114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.161570 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-dshm\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.164943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.164920 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tls-certs\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.209035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.208996 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gt4c\" (UniqueName: \"kubernetes.io/projected/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-kube-api-access-2gt4c\") pod \"premium-w1-4fe3a4-kserve-56cbbc644d-t258f\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.288075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.288035 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:04.685028 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:04.684998 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f"] Oct 31 16:16:04.691928 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:16:04.691895 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd43fd7b4_2fb6_49a4_9902_c1f7b5589784.slice/crio-b341b5a491537f057a06d3df29552e51fb376aefcda593532a30acc0acbd27bd WatchSource:0}: Error finding container b341b5a491537f057a06d3df29552e51fb376aefcda593532a30acc0acbd27bd: Status 404 returned error can't find the container with id b341b5a491537f057a06d3df29552e51fb376aefcda593532a30acc0acbd27bd Oct 31 16:16:05.319477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:05.319439 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" event={"ID":"d43fd7b4-2fb6-49a4-9902-c1f7b5589784","Type":"ContainerStarted","Data":"e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3"} Oct 31 16:16:05.319477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:05.319488 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" event={"ID":"d43fd7b4-2fb6-49a4-9902-c1f7b5589784","Type":"ContainerStarted","Data":"b341b5a491537f057a06d3df29552e51fb376aefcda593532a30acc0acbd27bd"} Oct 31 16:16:05.426509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:05.426452 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podStartSLOduration=2.426437097 podStartE2EDuration="2.426437097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:05.424315685 +0000 UTC m=+812.459610468" watchObservedRunningTime="2026-10-31 16:16:05.426437097 +0000 UTC m=+812.461731901" Oct 31 16:16:06.288417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:06.288386 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:08.668653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:08.668620 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:13.458860 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:13.458824 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:14.288859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:14.288821 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:14.303070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:14.303043 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:14.431005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:14.430969 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:16:18.360918 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:18.360884 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:20.395940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:20.395896 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:23.155621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:23.155587 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:27.751293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:27.751247 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:30.022676 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:30.022639 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:32.952017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:32.951980 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:33.880928 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:16:33.880894 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:16:37.754699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:37.754665 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:41.748560 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:41.748517 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:42.579606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:42.579573 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:47.364385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:47.364345 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:52.291532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:52.291497 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:16:57.040377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:16:57.040340 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:01.854537 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:01.854501 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:03.882078 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:17:03.882039 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:17:06.764221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:06.764185 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:11.664449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:11.664413 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:16.757412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:16.757371 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:21.633514 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:21.633474 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:23.856477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:23.856442 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:26.456000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:26.455957 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:28.089206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:28.089172 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:31.438089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:31.438055 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:33.197170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:33.197115 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:17:33.199312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:33.199293 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:17:33.204524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:33.204505 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:17:33.206682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:33.206662 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:17:33.215159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:33.215118 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:17:33.217325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:33.217307 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:17:33.881233 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:17:33.881194 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:17:36.268250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:36.268212 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:41.054574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:41.054540 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:45.877217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:45.877176 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:50.677684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:50.677648 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:52.882328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:52.882289 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:55.462106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:55.462072 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:17:58.544776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:17:58.544744 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:00.346429 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:00.346390 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:02.615759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:02.615717 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:03.881662 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:18:03.881628 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:18:04.944448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:04.944415 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7"] Oct 31 16:18:04.948810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:04.948788 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019157 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtzw\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-kube-api-access-pbtzw\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019206 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019253 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019274 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019379 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019450 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019493 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff17b788-4ed8-4981-bf66-7439999a184e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019542 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff17b788-4ed8-4981-bf66-7439999a184e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.019618 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.019561 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.047103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.047071 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7"] Oct 31 16:18:05.120338 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120306 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120385 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff17b788-4ed8-4981-bf66-7439999a184e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120422 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff17b788-4ed8-4981-bf66-7439999a184e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120447 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120499 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtzw\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-kube-api-access-pbtzw\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120528 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120598 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120631 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120686 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.120948 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120929 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.121012 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.120981 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.121260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.121235 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff17b788-4ed8-4981-bf66-7439999a184e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.121343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.121267 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.121343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.121298 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.122800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.122776 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.122919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.122899 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff17b788-4ed8-4981-bf66-7439999a184e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.163288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.163250 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:05.172876 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.172852 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.173004 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.172933 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtzw\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-kube-api-access-pbtzw\") pod \"maas-default-gateway-openshift-default-fcb75c45f-zvlv7\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.262510 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.262370 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:05.691610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.691575 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7"] Oct 31 16:18:05.697579 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:18:05.697539 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff17b788_4ed8_4981_bf66_7439999a184e.slice/crio-fb1f08f443bcdebe434c9bcf844cb414b227915cb80a96db5c1663fda80e26b4 WatchSource:0}: Error finding container fb1f08f443bcdebe434c9bcf844cb414b227915cb80a96db5c1663fda80e26b4: Status 404 returned error can't find the container with id fb1f08f443bcdebe434c9bcf844cb414b227915cb80a96db5c1663fda80e26b4 Oct 31 16:18:05.699942 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.699908 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:05.700051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.700002 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:05.700051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:05.700046 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:06.119237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:06.119121 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" event={"ID":"ff17b788-4ed8-4981-bf66-7439999a184e","Type":"ContainerStarted","Data":"9b894f0dcca49e3c56cb89a0121ad888fdc2435a3000dffbd855a7f1011510d8"} Oct 31 16:18:06.119237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:06.119218 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" event={"ID":"ff17b788-4ed8-4981-bf66-7439999a184e","Type":"ContainerStarted","Data":"fb1f08f443bcdebe434c9bcf844cb414b227915cb80a96db5c1663fda80e26b4"} Oct 31 16:18:06.263201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:06.263162 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:06.265023 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:06.264995 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zvlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 16:18:06.265187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:06.265047 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 16:18:06.278554 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:06.278500 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podStartSLOduration=2.278487376 podStartE2EDuration="2.278487376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:06.277542552 +0000 UTC m=+933.312837357" watchObservedRunningTime="2026-10-31 16:18:06.278487376 +0000 UTC m=+933.313782179" Oct 31 16:18:07.263547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:07.263510 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zvlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 16:18:07.263945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:07.263569 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 16:18:08.262874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:08.262836 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zvlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 16:18:08.263062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:08.262899 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 16:18:09.263071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:09.263028 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zvlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 16:18:09.263639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:09.263090 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 16:18:10.177260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:10.177229 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:10.263667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:10.263634 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zvlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 16:18:10.264054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:10.263690 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 16:18:11.263728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:11.263689 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zvlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 16:18:11.264116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:11.263742 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 16:18:12.262715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:12.262679 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-zvlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 16:18:12.262860 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:12.262742 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 16:18:13.967325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:13.967237 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:13.967826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:13.967806 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:13.968705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:13.968685 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:18:15.046664 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:15.046632 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:18.142119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.142081 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx"] Oct 31 16:18:18.145810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.145791 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.171822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.171797 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.171934 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.171834 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.171934 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.171928 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2sk\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-kube-api-access-pz2sk\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.172049 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.172012 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.172100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.172073 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.172222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.172207 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.172266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.172240 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c69a68-e687-4404-a29a-f5c441377289-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.172318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.172267 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c69a68-e687-4404-a29a-f5c441377289-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.172391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.172374 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.194236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.194211 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a1685e-openshift-default-dockercfg-2cr54\"" Oct 31 16:18:18.244174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.244121 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx"] Oct 31 16:18:18.273885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.273857 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.273898 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c69a68-e687-4404-a29a-f5c441377289-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.273927 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c69a68-e687-4404-a29a-f5c441377289-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.273948 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274074 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274118 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274239 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2sk\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-kube-api-access-pz2sk\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274292 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274353 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274411 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274459 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274917 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274668 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274917 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274709 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.274917 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.274899 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c69a68-e687-4404-a29a-f5c441377289-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.276765 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.276717 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.277284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.276902 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c69a68-e687-4404-a29a-f5c441377289-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.326955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.326927 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.327112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.327091 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2sk\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-kube-api-access-pz2sk\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.460255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.460169 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:18.757532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.757450 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j"] Oct 31 16:18:18.762319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.762298 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.815469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.815440 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j"] Oct 31 16:18:18.881342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881306 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881337 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881581 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881481 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881581 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881545 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdkr\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-kube-api-access-qvdkr\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881646 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881702 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881732 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881799 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.881858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.881842 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.884957 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.884936 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx"] Oct 31 16:18:18.891617 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:18:18.891593 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c69a68_e687_4404_a29a_f5c441377289.slice/crio-e3fcbf7dce1b6a15725d26d0e6212c3e98a266783d5ef35a18bc121262d588da WatchSource:0}: Error finding container e3fcbf7dce1b6a15725d26d0e6212c3e98a266783d5ef35a18bc121262d588da: Status 404 returned error can't find the container with id e3fcbf7dce1b6a15725d26d0e6212c3e98a266783d5ef35a18bc121262d588da Oct 31 16:18:18.893900 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.893872 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:18.893961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.893930 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:18.894006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.893960 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:18.982937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.982902 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.982952 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983009 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983034 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983108 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983147 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983202 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983251 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdkr\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-kube-api-access-qvdkr\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983496 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983316 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983496 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983422 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983588 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983632 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.983735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.983702 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.984180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.984127 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.985817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.985793 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:18.986042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:18.986022 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:19.039785 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.039704 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:19.040044 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.040016 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdkr\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-kube-api-access-qvdkr\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-8569j\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:19.079423 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.079395 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:19.244772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.244726 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" event={"ID":"a3c69a68-e687-4404-a29a-f5c441377289","Type":"ContainerStarted","Data":"245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95"} Oct 31 16:18:19.245207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.244793 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" event={"ID":"a3c69a68-e687-4404-a29a-f5c441377289","Type":"ContainerStarted","Data":"e3fcbf7dce1b6a15725d26d0e6212c3e98a266783d5ef35a18bc121262d588da"} Oct 31 16:18:19.353670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.353563 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podStartSLOduration=1.353549095 podStartE2EDuration="1.353549095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:19.351320975 +0000 UTC m=+946.386615778" watchObservedRunningTime="2026-10-31 16:18:19.353549095 +0000 UTC m=+946.388843923" Oct 31 16:18:19.461366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.461320 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:19.463322 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.463290 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 16:18:19.463454 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.463348 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 16:18:19.509890 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.509851 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j"] Oct 31 16:18:19.515467 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:18:19.515435 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc87830_f90e_40d8_a712_b0ecd2cbcd94.slice/crio-97d03e6115446371472931ee17678211b924eab33c7c2f7e265eeb63fd6cb492 WatchSource:0}: Error finding container 97d03e6115446371472931ee17678211b924eab33c7c2f7e265eeb63fd6cb492: Status 404 returned error can't find the container with id 97d03e6115446371472931ee17678211b924eab33c7c2f7e265eeb63fd6cb492 Oct 31 16:18:19.518533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.518498 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:19.518622 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.518570 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:19.518622 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.518599 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:19.872056 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.871963 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:19.966790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.966756 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw"] Oct 31 16:18:19.970931 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:19.970908 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.060345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.060309 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw"] Oct 31 16:18:20.095459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095425 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657dt\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-kube-api-access-657dt\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.095744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095537 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.095744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095630 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.095744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095662 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.095744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095693 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.096002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095792 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.096002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095858 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.096002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095948 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.096267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.095996 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.196928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.196893 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.196928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.196932 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197202 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.196960 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197202 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197078 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197219 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197270 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197408 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197309 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657dt\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-kube-api-access-657dt\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197408 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197392 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197423 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197441 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197460 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197737 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197712 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.197845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.197760 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.198020 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.198001 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.199875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.199851 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.200063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.200042 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.250389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.250299 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657dt\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-kube-api-access-657dt\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.250389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.250304 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-vkxlw\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.254804 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.254774 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" event={"ID":"8bc87830-f90e-40d8-a712-b0ecd2cbcd94","Type":"ContainerStarted","Data":"3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055"} Oct 31 16:18:20.254922 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.254822 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" event={"ID":"8bc87830-f90e-40d8-a712-b0ecd2cbcd94","Type":"ContainerStarted","Data":"97d03e6115446371472931ee17678211b924eab33c7c2f7e265eeb63fd6cb492"} Oct 31 16:18:20.286104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.286035 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:20.365431 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.365375 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podStartSLOduration=2.365359328 podStartE2EDuration="2.365359328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:20.361797035 +0000 UTC m=+947.397091843" watchObservedRunningTime="2026-10-31 16:18:20.365359328 +0000 UTC m=+947.400654133" Oct 31 16:18:20.461441 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.461354 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 16:18:20.461441 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.461403 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 16:18:20.763223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.753701 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw"] Oct 31 16:18:20.777179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.777116 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:20.777266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.777214 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:20.777266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:20.777245 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:21.009433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.009379 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:21.080074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.079989 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:21.081320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.081292 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 16:18:21.081439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.081338 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 16:18:21.281412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.281379 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" event={"ID":"3d4c9404-b8ae-4922-8e86-1d90533df8d9","Type":"ContainerStarted","Data":"cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0"} Oct 31 16:18:21.281978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.281954 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" event={"ID":"3d4c9404-b8ae-4922-8e86-1d90533df8d9","Type":"ContainerStarted","Data":"1abc4713bdfda510a47908de0c5727724a818cb04dd6701bad61b9682d8245db"} Oct 31 16:18:21.286520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.286496 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:21.288730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.288706 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:21.288856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.288750 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:21.406886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.406822 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podStartSLOduration=2.406802856 podStartE2EDuration="2.406802856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:21.401328512 +0000 UTC m=+948.436623317" watchObservedRunningTime="2026-10-31 16:18:21.406802856 +0000 UTC m=+948.442097661" Oct 31 16:18:21.466075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.465532 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 16:18:21.466075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:21.465591 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 16:18:22.081164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:22.080549 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 16:18:22.081164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:22.080627 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 16:18:22.293168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:22.289534 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:22.293168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:22.289595 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:22.463160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:22.462550 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 16:18:22.463160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:22.462614 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 16:18:23.088162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:23.083540 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 16:18:23.088162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:23.083604 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 16:18:23.296162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:23.295527 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:23.296162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:23.295594 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:23.470580 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:23.470536 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 16:18:23.485200 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:23.478548 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 16:18:24.081595 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:24.081552 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 16:18:24.081883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:24.081854 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 16:18:24.287160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:24.286572 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:24.287160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:24.286639 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:24.471160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:24.470515 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 16:18:24.471160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:24.470572 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 16:18:25.080599 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:25.079695 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 16:18:25.080599 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:25.079745 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 16:18:25.101336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:25.101243 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:25.290187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:25.286916 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:25.290187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:25.286969 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:25.465164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:25.464540 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 31 16:18:25.465164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:25.464604 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 31 16:18:26.080162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:26.079772 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 16:18:26.080162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:26.079839 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 16:18:26.286909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:26.286864 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:26.287110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:26.286921 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:26.465290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:26.465255 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:26.465646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:26.465626 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:26.466467 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:26.466444 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:18:27.098078 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:27.098051 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:27.099072 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:27.099046 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:27.100384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:27.100364 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:18:27.286518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:27.286477 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:27.286682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:27.286555 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:28.286843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:28.286801 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-vkxlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 16:18:28.287339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:28.286869 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 16:18:28.413868 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:28.413833 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:30.028346 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:30.028310 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:30.028823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:30.028748 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:30.029700 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:30.029682 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:18:30.079419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:30.079387 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:33.881075 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:18:33.881034 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:18:34.864235 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:34.864204 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:39.778902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:39.778862 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:44.550012 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:44.549978 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:48.122411 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.122378 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7"] Oct 31 16:18:48.126270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.126248 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.178679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.178644 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7"] Oct 31 16:18:48.185979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.185948 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vkn\" (UniqueName: \"kubernetes.io/projected/af2f32e8-885f-4eae-8301-54da9247f464-kube-api-access-q4vkn\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.186127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.186018 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-home\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.186243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.186221 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2f32e8-885f-4eae-8301-54da9247f464-tls-certs\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.186425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.186401 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-tmp-dir\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.186477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.186443 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-model-cache\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.186880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.186862 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-dshm\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.220958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.220934 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"kube-root-ca.crt\"" Oct 31 16:18:48.221169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.221149 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"openshift-service-ca.crt\"" Oct 31 16:18:48.221390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.221374 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"default-dockercfg-zbwr8\"" Oct 31 16:18:48.221653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.221636 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-59357a\"/\"distinct-w4-59357a-kserve-self-signed-certs\"" Oct 31 16:18:48.223833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.223809 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:48.288311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288277 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-home\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.288478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288356 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2f32e8-885f-4eae-8301-54da9247f464-tls-certs\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.288478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288444 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-tmp-dir\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.288478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288468 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-model-cache\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.288588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288544 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-dshm\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.288650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288633 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vkn\" (UniqueName: \"kubernetes.io/projected/af2f32e8-885f-4eae-8301-54da9247f464-kube-api-access-q4vkn\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.288849 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288782 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-tmp-dir\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.288849 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288786 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-home\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.289012 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.288890 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-model-cache\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.290784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.290757 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-dshm\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.290942 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.290926 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2f32e8-885f-4eae-8301-54da9247f464-tls-certs\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.341904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.341873 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vkn\" (UniqueName: \"kubernetes.io/projected/af2f32e8-885f-4eae-8301-54da9247f464-kube-api-access-q4vkn\") pod \"distinct-w4-59357a-kserve-5bb9cc867-bv8x7\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.438785 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.438747 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:48.849429 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:48.849330 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7"] Oct 31 16:18:48.854798 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:18:48.854768 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf2f32e8_885f_4eae_8301_54da9247f464.slice/crio-dd8d3fb680331caeaeeff780af1f19cc7db5219f5845f08a9df8d88f968a5b22 WatchSource:0}: Error finding container dd8d3fb680331caeaeeff780af1f19cc7db5219f5845f08a9df8d88f968a5b22: Status 404 returned error can't find the container with id dd8d3fb680331caeaeeff780af1f19cc7db5219f5845f08a9df8d88f968a5b22 Oct 31 16:18:49.374815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:49.374782 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:49.532507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:49.532471 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" event={"ID":"af2f32e8-885f-4eae-8301-54da9247f464","Type":"ContainerStarted","Data":"ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8"} Oct 31 16:18:49.532669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:49.532518 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" event={"ID":"af2f32e8-885f-4eae-8301-54da9247f464","Type":"ContainerStarted","Data":"dd8d3fb680331caeaeeff780af1f19cc7db5219f5845f08a9df8d88f968a5b22"} Oct 31 16:18:49.641803 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:49.641742 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podStartSLOduration=1.641728012 podStartE2EDuration="1.641728012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:49.639408931 +0000 UTC m=+976.674703736" watchObservedRunningTime="2026-10-31 16:18:49.641728012 +0000 UTC m=+976.677022817" Oct 31 16:18:49.922469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:49.922386 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42"] Oct 31 16:18:49.926459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:49.926437 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:49.980310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:49.980275 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ef6d93-openshift-default-dockercfg-dzsjw\"" Oct 31 16:18:50.011043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011007 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011121 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011185 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011294 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011326 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8tj9\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-kube-api-access-c8tj9\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011372 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011414 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011440 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.011574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.011479 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.029863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.029832 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42"] Oct 31 16:18:50.112886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.112848 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.112886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.112891 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.112909 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.112982 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113004 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8tj9\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-kube-api-access-c8tj9\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113036 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113078 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113111 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113183 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113363 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113450 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113533 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.113740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.113723 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.114084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.114061 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.115325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.115302 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.115964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.115946 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.166264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.166232 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8tj9\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-kube-api-access-c8tj9\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.166412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.166227 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.240731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.240634 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:50.439471 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.439436 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:50.684303 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.684268 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42"] Oct 31 16:18:50.732642 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:18:50.732604 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14974db_dd36_4395_b4cc_e3ebbd7bbe16.slice/crio-49ae6e8ed970745d2223d510bde46fe764637ef5aa610fb8832a46cfadbf14ec WatchSource:0}: Error finding container 49ae6e8ed970745d2223d510bde46fe764637ef5aa610fb8832a46cfadbf14ec: Status 404 returned error can't find the container with id 49ae6e8ed970745d2223d510bde46fe764637ef5aa610fb8832a46cfadbf14ec Oct 31 16:18:50.735102 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.735060 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:50.735238 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.735191 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:50.735305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:50.735238 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:18:51.554404 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:51.554363 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" event={"ID":"c14974db-dd36-4395-b4cc-e3ebbd7bbe16","Type":"ContainerStarted","Data":"39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765"} Oct 31 16:18:51.554870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:51.554423 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" event={"ID":"c14974db-dd36-4395-b4cc-e3ebbd7bbe16","Type":"ContainerStarted","Data":"49ae6e8ed970745d2223d510bde46fe764637ef5aa610fb8832a46cfadbf14ec"} Oct 31 16:18:51.757515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:51.757464 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podStartSLOduration=2.757448375 podStartE2EDuration="2.757448375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:51.70865089 +0000 UTC m=+978.743945695" watchObservedRunningTime="2026-10-31 16:18:51.757448375 +0000 UTC m=+978.792743190" Oct 31 16:18:52.243867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:52.243832 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:52.244219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:52.244194 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 16:18:52.244422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:52.244399 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 16:18:53.242162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:53.241522 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 16:18:53.242162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:53.241587 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 16:18:54.189148 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:54.189107 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:54.246159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:54.245515 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 16:18:54.246159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:54.245578 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 16:18:55.245540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:55.245498 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 16:18:55.245753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:55.245553 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 16:18:56.242157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:56.241504 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 16:18:56.242157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:56.241567 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 16:18:57.089928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.089895 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f"] Oct 31 16:18:57.094796 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.094778 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.140453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.140418 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f"] Oct 31 16:18:57.181277 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.181240 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"kube-root-ca.crt\"" Oct 31 16:18:57.181450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.181431 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"openshift-service-ca.crt\"" Oct 31 16:18:57.181519 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.181483 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"default-dockercfg-4f4mn\"" Oct 31 16:18:57.181915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.181893 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:57.189302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.189285 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ac1273\"/\"distinct-w0-ac1273-kserve-self-signed-certs\"" Oct 31 16:18:57.213132 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.213110 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-dshm\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.213254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.213177 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2hn\" (UniqueName: \"kubernetes.io/projected/f250b3cf-92a0-421e-a50f-710da539a867-kube-api-access-md2hn\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.213396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.213378 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f250b3cf-92a0-421e-a50f-710da539a867-tls-certs\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.213484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.213415 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-tmp-dir\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.213587 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.213564 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-home\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.213644 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.213603 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-model-cache\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.241644 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.241616 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 16:18:57.241799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.241668 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 16:18:57.314228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314196 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-home\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.314712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314236 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-model-cache\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.314712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314295 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-dshm\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.314712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314316 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2hn\" (UniqueName: \"kubernetes.io/projected/f250b3cf-92a0-421e-a50f-710da539a867-kube-api-access-md2hn\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.314712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314430 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f250b3cf-92a0-421e-a50f-710da539a867-tls-certs\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.314712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314456 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-tmp-dir\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.314712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314670 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-home\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.315038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314850 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-tmp-dir\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.315038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.314897 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-model-cache\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.316890 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.316860 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-dshm\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.317205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.317186 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f250b3cf-92a0-421e-a50f-710da539a867-tls-certs\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.367723 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.367645 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2hn\" (UniqueName: \"kubernetes.io/projected/f250b3cf-92a0-421e-a50f-710da539a867-kube-api-access-md2hn\") pod \"distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.419522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.419488 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:18:57.911269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:57.911240 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f"] Oct 31 16:18:57.916169 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:18:57.916125 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf250b3cf_92a0_421e_a50f_710da539a867.slice/crio-8d13c6e329565664e113bb83d4c65654c9aac940f4be0ec712874125dcafe560 WatchSource:0}: Error finding container 8d13c6e329565664e113bb83d4c65654c9aac940f4be0ec712874125dcafe560: Status 404 returned error can't find the container with id 8d13c6e329565664e113bb83d4c65654c9aac940f4be0ec712874125dcafe560 Oct 31 16:18:58.245752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.245672 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:58.246092 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.246071 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:58.246882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.246863 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:18:58.439219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.439177 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:58.452103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.452072 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:58.643487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.643452 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" event={"ID":"f250b3cf-92a0-421e-a50f-710da539a867","Type":"ContainerStarted","Data":"6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66"} Oct 31 16:18:58.643646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.643497 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" event={"ID":"f250b3cf-92a0-421e-a50f-710da539a867","Type":"ContainerStarted","Data":"8d13c6e329565664e113bb83d4c65654c9aac940f4be0ec712874125dcafe560"} Oct 31 16:18:58.655097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.655067 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:18:58.748323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:58.748267 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podStartSLOduration=1.748252614 podStartE2EDuration="1.748252614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:18:58.747303345 +0000 UTC m=+985.782598154" watchObservedRunningTime="2026-10-31 16:18:58.748252614 +0000 UTC m=+985.783547418" Oct 31 16:18:59.057688 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:59.057598 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:18:59.420076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:18:59.420042 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:19:03.884372 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:19:03.884329 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:19:03.943752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:03.943708 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:07.419664 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:07.419634 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:19:07.432267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:07.432243 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:19:07.549146 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:07.549105 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:07.719757 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:07.719676 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:19:08.843895 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:08.843859 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:13.343476 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:13.343440 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:13.667089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:13.667049 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:18.566064 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:18.566023 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:21.237043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.237009 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td"] Oct 31 16:19:21.241378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.241356 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.294247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.294214 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td"] Oct 31 16:19:21.311771 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.311739 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fca131-9d77-45bf-ade0-43efe5862df3-tls-certs\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.311947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.311857 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txq56\" (UniqueName: \"kubernetes.io/projected/97fca131-9d77-45bf-ade0-43efe5862df3-kube-api-access-txq56\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.311947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.311900 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-model-cache\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.312049 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.312031 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-tmp-dir\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.312087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.312070 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-dshm\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.312172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.312088 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-home\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.335274 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.335249 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"distinct-w6-a1685e-kserve-self-signed-certs\"" Oct 31 16:19:21.341681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.341655 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:21.413076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413044 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txq56\" (UniqueName: \"kubernetes.io/projected/97fca131-9d77-45bf-ade0-43efe5862df3-kube-api-access-txq56\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.413295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413085 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-model-cache\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.413295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413192 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-tmp-dir\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.413295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413217 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-dshm\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.413295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413261 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-home\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.413520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413394 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fca131-9d77-45bf-ade0-43efe5862df3-tls-certs\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.414229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413582 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-model-cache\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.414229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413765 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-tmp-dir\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.414229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.413970 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-home\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.415766 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.415734 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-dshm\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.416122 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.416106 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fca131-9d77-45bf-ade0-43efe5862df3-tls-certs\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.465858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.465829 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txq56\" (UniqueName: \"kubernetes.io/projected/97fca131-9d77-45bf-ade0-43efe5862df3-kube-api-access-txq56\") pod \"distinct-w6-a1685e-kserve-58c9bd959c-hd7td\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.553949 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.553867 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:21.943672 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:21.943637 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td"] Oct 31 16:19:21.949029 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:19:21.948999 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97fca131_9d77_45bf_ade0_43efe5862df3.slice/crio-0a301013a09000bfae9c1fb2c017997c98005e3753a509d7ed94105772c6139c WatchSource:0}: Error finding container 0a301013a09000bfae9c1fb2c017997c98005e3753a509d7ed94105772c6139c: Status 404 returned error can't find the container with id 0a301013a09000bfae9c1fb2c017997c98005e3753a509d7ed94105772c6139c Oct 31 16:19:22.825341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:22.825308 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" event={"ID":"97fca131-9d77-45bf-ade0-43efe5862df3","Type":"ContainerStarted","Data":"6a8a41ca613e0ad1c80d9bdcda68e21b74473cace7a4ef6bf188c1fb3fb0d5f4"} Oct 31 16:19:22.825341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:22.825352 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" event={"ID":"97fca131-9d77-45bf-ade0-43efe5862df3","Type":"ContainerStarted","Data":"0a301013a09000bfae9c1fb2c017997c98005e3753a509d7ed94105772c6139c"} Oct 31 16:19:22.932992 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:22.932936 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podStartSLOduration=1.932920162 podStartE2EDuration="1.932920162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:22.930128506 +0000 UTC m=+1009.965423311" watchObservedRunningTime="2026-10-31 16:19:22.932920162 +0000 UTC m=+1009.968214967" Oct 31 16:19:23.366198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:23.366165 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:23.554563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:23.554519 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:28.127229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:28.127185 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:28.753346 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:28.753313 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:31.554063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:31.554009 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:31.568687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:31.568659 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:31.903547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:31.903518 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:19:33.045262 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:33.045217 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:33.884741 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:19:33.884705 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:19:34.366631 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.366553 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq"] Oct 31 16:19:34.371607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.371585 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.471173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.471130 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq"] Oct 31 16:19:34.476554 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476526 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.476699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476570 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.476699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476660 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcp8g\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-kube-api-access-pcp8g\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.476804 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476710 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.476804 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476786 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.476899 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476825 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.476951 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476904 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.476951 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.476941 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.477183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.477131 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.578756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.578708 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.578756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.578762 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.578812 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.578860 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.578889 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.578915 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcp8g\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-kube-api-access-pcp8g\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579048 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579165 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579217 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579254 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-data\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579539 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-certs\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579691 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579634 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.579749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579695 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-credential-socket\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.580009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.579984 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istiod-ca-cert\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.581523 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.581500 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-podinfo\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.581650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.581563 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-envoy\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.626766 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.626691 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf"] Oct 31 16:19:34.631421 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.631398 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.631930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.631908 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-token\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.632041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.632024 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcp8g\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-kube-api-access-pcp8g\") pod \"e2e-worker-w0-ac1273-openshift-default-79699df87-stllq\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.681319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.681293 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-59357a-openshift-default-dockercfg-z7nvh\"" Oct 31 16:19:34.684584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.684556 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:34.686426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.686400 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf"] Oct 31 16:19:34.783013 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.782754 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783013 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.782862 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783013 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.782927 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.783094 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.783152 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a52591-07b2-491e-a873-a537c7278595-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.783234 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jq6\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-kube-api-access-p2jq6\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.783342 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.783366 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a52591-07b2-491e-a873-a537c7278595-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.783450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.783423 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.884734 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.884708 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.884914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.884741 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a52591-07b2-491e-a873-a537c7278595-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.884914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.884802 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jq6\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-kube-api-access-p2jq6\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.884914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.884847 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.884914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.884870 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a52591-07b2-491e-a873-a537c7278595-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.884906 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.884974 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.885051 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.885091 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.885223 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.885339 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.885511 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.885638 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a52591-07b2-491e-a873-a537c7278595-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.885809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.885753 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.887854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.887833 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.888105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.888079 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a52591-07b2-491e-a873-a537c7278595-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.937859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.937827 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.938018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.937952 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jq6\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-kube-api-access-p2jq6\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:34.964850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:34.964825 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:35.136726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.136634 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq"] Oct 31 16:19:35.141267 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:19:35.141235 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeab96e1_8cff_43b4_be2c_6a0c3b64348d.slice/crio-0a46b7af1c119abce0bb21c770080571bdda498407d1ab6a4127bc8da53a6299 WatchSource:0}: Error finding container 0a46b7af1c119abce0bb21c770080571bdda498407d1ab6a4127bc8da53a6299: Status 404 returned error can't find the container with id 0a46b7af1c119abce0bb21c770080571bdda498407d1ab6a4127bc8da53a6299 Oct 31 16:19:35.143948 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.143908 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:35.144039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.144004 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:35.144077 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.144064 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:35.344801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.344769 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf"] Oct 31 16:19:35.349218 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:19:35.349195 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a52591_07b2_491e_a873_a537c7278595.slice/crio-ae8c474d082a5d5849804e39f8635a8c601f4c062acd8b10001b0e30f3d14a6b WatchSource:0}: Error finding container ae8c474d082a5d5849804e39f8635a8c601f4c062acd8b10001b0e30f3d14a6b: Status 404 returned error can't find the container with id ae8c474d082a5d5849804e39f8635a8c601f4c062acd8b10001b0e30f3d14a6b Oct 31 16:19:35.351871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.351840 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:35.351998 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.351940 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:35.352063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.352002 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:35.671725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.671615 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:35.925292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.925200 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" event={"ID":"deab96e1-8cff-43b4-be2c-6a0c3b64348d","Type":"ContainerStarted","Data":"0cc30f6e9aba11e9f51d630633f76b373e392fc2d7385c75dc9b7f71318a81a0"} Oct 31 16:19:35.927064 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.927034 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" event={"ID":"deab96e1-8cff-43b4-be2c-6a0c3b64348d","Type":"ContainerStarted","Data":"0a46b7af1c119abce0bb21c770080571bdda498407d1ab6a4127bc8da53a6299"} Oct 31 16:19:35.928085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.928061 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" event={"ID":"b5a52591-07b2-491e-a873-a537c7278595","Type":"ContainerStarted","Data":"86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e"} Oct 31 16:19:35.928374 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.928356 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" event={"ID":"b5a52591-07b2-491e-a873-a537c7278595","Type":"ContainerStarted","Data":"ae8c474d082a5d5849804e39f8635a8c601f4c062acd8b10001b0e30f3d14a6b"} Oct 31 16:19:35.968287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:35.968250 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:36.056169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:36.055813 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podStartSLOduration=2.055788683 podStartE2EDuration="2.055788683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:36.036428463 +0000 UTC m=+1023.071723267" watchObservedRunningTime="2026-10-31 16:19:36.055788683 +0000 UTC m=+1023.091083490" Oct 31 16:19:36.147160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:36.146296 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podStartSLOduration=2.146276175 podStartE2EDuration="2.146276175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:36.143703167 +0000 UTC m=+1023.178997988" watchObservedRunningTime="2026-10-31 16:19:36.146276175 +0000 UTC m=+1023.181570981" Oct 31 16:19:36.685494 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:36.685461 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:36.686841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:36.686819 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 16:19:36.686906 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:36.686868 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 16:19:36.973833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:36.973746 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 16:19:36.973833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:36.973796 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 16:19:37.685402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:37.685361 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 16:19:37.685608 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:37.685420 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 16:19:37.971620 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:37.971530 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 16:19:37.971861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:37.971835 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 16:19:38.056162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:38.055568 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:38.685761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:38.685729 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 16:19:38.686162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:38.685777 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 16:19:38.966015 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:38.965514 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 16:19:38.966015 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:38.965577 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 16:19:39.685755 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:39.685718 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 16:19:39.685963 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:39.685779 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 16:19:39.969684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:39.969585 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 16:19:39.969972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:39.969942 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 16:19:40.685298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:40.685263 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 16:19:40.685471 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:40.685329 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 16:19:40.967216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:40.966515 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 16:19:40.967216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:40.966575 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 16:19:41.684865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:41.684828 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 16:19:41.685032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:41.684890 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 16:19:41.965567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:41.965476 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 16:19:41.965567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:41.965541 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 16:19:42.689809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:42.689778 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:42.690369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:42.690080 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:42.690901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:42.690882 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:19:42.972334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:42.972241 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:42.973631 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:42.973471 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:42.974074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:42.974051 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:19:43.041543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:43.041512 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:47.845233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:47.845193 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:48.511993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.511959 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd"] Oct 31 16:19:48.516003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.515978 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.564473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.564446 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4fe3a4-openshift-default-dockercfg-njqjl\"" Oct 31 16:19:48.598530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598500 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.598629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598557 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.598720 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598698 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.598773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598758 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.598807 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598789 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.598840 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598816 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.598942 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598926 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.598977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598950 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.599009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.598979 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttfd\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-kube-api-access-sttfd\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.614371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.614335 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd"] Oct 31 16:19:48.699678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699652 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.699678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699681 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.699891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699713 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttfd\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-kube-api-access-sttfd\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.699891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699744 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.699891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699798 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.699891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699883 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699929 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699957 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.699985 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.700124 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.700184 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.700327 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.700374 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.700924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.700901 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.702201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.702176 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.702549 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.702533 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.752293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.752265 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.752400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.752386 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttfd\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-kube-api-access-sttfd\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.829979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.829910 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:48.837177 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.837126 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz"] Oct 31 16:19:48.845491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.845469 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.900837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.898960 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz"] Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.907822 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4180fbb7-3252-40b1-b401-34b666278d57-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.907946 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ng4\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-kube-api-access-z9ng4\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.908031 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.908093 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.908252 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4180fbb7-3252-40b1-b401-34b666278d57-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.908296 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.908345 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.908372 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:48.909240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:48.908430 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010122 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010089 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010176 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4180fbb7-3252-40b1-b401-34b666278d57-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010202 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ng4\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-kube-api-access-z9ng4\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010266 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010445 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010541 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4180fbb7-3252-40b1-b401-34b666278d57-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010572 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010608 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010626 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010690 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.010873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010828 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.011017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.010994 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.011062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.011039 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.011277 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.011256 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4180fbb7-3252-40b1-b401-34b666278d57-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.012710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.012692 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4180fbb7-3252-40b1-b401-34b666278d57-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.012795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.012753 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.063656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.063635 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.063897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.063876 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ng4\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-kube-api-access-z9ng4\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.080094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.080045 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl"] Oct 31 16:19:49.085093 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.085039 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.137068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.136452 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f7821a-openshift-default-dockercfg-pbwpd\"" Oct 31 16:19:49.143176 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.141065 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl"] Oct 31 16:19:49.165635 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.165608 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:49.213317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.213280 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.213459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.213326 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.213459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.213391 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.214105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.213837 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.214105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.213942 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.214105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.213976 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.214105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.214052 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8299\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-kube-api-access-m8299\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.214451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.214307 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.214451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.214386 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.257862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.257832 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd"] Oct 31 16:19:49.261776 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:19:49.261729 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a05bcb5_0ec3_42ea_8c14_0e100ea8943b.slice/crio-725feeccd29ece5e952d58a02d93ecf33535c320977d923c2589bf3f8f8086c0 WatchSource:0}: Error finding container 725feeccd29ece5e952d58a02d93ecf33535c320977d923c2589bf3f8f8086c0: Status 404 returned error can't find the container with id 725feeccd29ece5e952d58a02d93ecf33535c320977d923c2589bf3f8f8086c0 Oct 31 16:19:49.274343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.274303 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.274451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.274414 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.274491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.274455 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.315389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315360 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.315489 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315403 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.315489 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315445 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315620 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315701 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315705 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315730 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315765 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8299\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-kube-api-access-m8299\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315817 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.315849 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.316102 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.316382 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.316626 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.316857 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.318482 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.319167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.318529 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.368081 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.368050 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.368480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.368462 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8299\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-kube-api-access-m8299\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.399282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.399247 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:49.544173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.544117 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz"] Oct 31 16:19:49.554170 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:19:49.549401 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4180fbb7_3252_40b1_b401_34b666278d57.slice/crio-b5931e8d5f8621b0986562a401b2d0b08c59c80cf470794b4a5c360a048fca15 WatchSource:0}: Error finding container b5931e8d5f8621b0986562a401b2d0b08c59c80cf470794b4a5c360a048fca15: Status 404 returned error can't find the container with id b5931e8d5f8621b0986562a401b2d0b08c59c80cf470794b4a5c360a048fca15 Oct 31 16:19:49.554170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.552528 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.554170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.552604 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.554170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.552640 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.785123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.785083 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl"] Oct 31 16:19:49.790671 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:19:49.790640 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf79a2f0_d2ac_45d2_8895_ffd676127f0d.slice/crio-92d13b8cd96a10e0841952d6821a4af8d732bfafa4d5b9190c3ae52fdfc5d3b8 WatchSource:0}: Error finding container 92d13b8cd96a10e0841952d6821a4af8d732bfafa4d5b9190c3ae52fdfc5d3b8: Status 404 returned error can't find the container with id 92d13b8cd96a10e0841952d6821a4af8d732bfafa4d5b9190c3ae52fdfc5d3b8 Oct 31 16:19:49.793389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.793357 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.793484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.793422 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:49.793484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:49.793448 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:19:50.063863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.063827 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" event={"ID":"4180fbb7-3252-40b1-b401-34b666278d57","Type":"ContainerStarted","Data":"38f528cd5b7c60c95f50988ff80d01e8f6f99daf99ba12676aa6c68893144a07"} Oct 31 16:19:50.064377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.063890 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" event={"ID":"4180fbb7-3252-40b1-b401-34b666278d57","Type":"ContainerStarted","Data":"b5931e8d5f8621b0986562a401b2d0b08c59c80cf470794b4a5c360a048fca15"} Oct 31 16:19:50.065658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.065631 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" event={"ID":"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b","Type":"ContainerStarted","Data":"e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653"} Oct 31 16:19:50.065794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.065683 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" event={"ID":"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b","Type":"ContainerStarted","Data":"725feeccd29ece5e952d58a02d93ecf33535c320977d923c2589bf3f8f8086c0"} Oct 31 16:19:50.067305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.067276 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" event={"ID":"bf79a2f0-d2ac-45d2-8895-ffd676127f0d","Type":"ContainerStarted","Data":"765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0"} Oct 31 16:19:50.067444 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.067325 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" event={"ID":"bf79a2f0-d2ac-45d2-8895-ffd676127f0d","Type":"ContainerStarted","Data":"92d13b8cd96a10e0841952d6821a4af8d732bfafa4d5b9190c3ae52fdfc5d3b8"} Oct 31 16:19:50.166167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.166092 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:50.168232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.168203 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 16:19:50.168399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.168256 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 16:19:50.176125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.175243 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podStartSLOduration=2.175225122 podStartE2EDuration="2.175225122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:50.17053016 +0000 UTC m=+1037.205824965" watchObservedRunningTime="2026-10-31 16:19:50.175225122 +0000 UTC m=+1037.210519941" Oct 31 16:19:50.289221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.286980 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podStartSLOduration=2.286961471 podStartE2EDuration="2.286961471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:50.281760011 +0000 UTC m=+1037.317054880" watchObservedRunningTime="2026-10-31 16:19:50.286961471 +0000 UTC m=+1037.322256276" Oct 31 16:19:50.399597 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.399563 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:50.401294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.401268 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 16:19:50.401436 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.401312 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 16:19:50.401990 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.401943 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podStartSLOduration=1.401926203 podStartE2EDuration="1.401926203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:19:50.397062865 +0000 UTC m=+1037.432357666" watchObservedRunningTime="2026-10-31 16:19:50.401926203 +0000 UTC m=+1037.437221032" Oct 31 16:19:50.840159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.832518 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:50.855161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.854279 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 16:19:50.855161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:50.854349 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 16:19:51.168548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:51.168510 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 16:19:51.169181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:51.169099 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 16:19:51.399672 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:51.399634 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 16:19:51.399875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:51.399697 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 16:19:51.839165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:51.838486 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 16:19:51.839165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:51.838547 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 16:19:52.167164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:52.166517 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 16:19:52.167164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:52.166577 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 16:19:52.405267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:52.404530 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 16:19:52.405267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:52.404592 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 16:19:52.837178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:52.833814 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 16:19:52.837178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:52.833888 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 16:19:52.933240 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:52.932628 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:53.183716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:53.183244 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 16:19:53.183716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:53.183305 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 16:19:53.400943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:53.400500 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 16:19:53.400943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:53.400563 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 16:19:53.833528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:53.833487 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 16:19:53.834123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:53.834089 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 16:19:54.008831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:54.008784 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:19:54.176540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:54.176500 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 16:19:54.176817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:54.176791 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 16:19:54.400492 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:54.400456 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 16:19:54.400739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:54.400714 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 16:19:54.832573 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:54.832534 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 16:19:54.832877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:54.832848 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 16:19:55.168482 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:55.168445 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 16:19:55.169055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:55.169025 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 16:19:55.400306 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:55.400273 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 16:19:55.400474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:55.400345 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 16:19:55.833532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:55.833494 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 16:19:55.833835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:55.833799 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 16:19:56.166690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:56.166651 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 16:19:56.166868 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:56.166719 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 16:19:56.400045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:56.400010 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 16:19:56.400426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:56.400074 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 16:19:57.242273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.240991 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:57.242273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.242204 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:57.245193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.244885 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:19:57.287639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.287608 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:57.288297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.288274 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:57.289895 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.289848 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:19:57.952039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.952003 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:57.953167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.952969 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:57.954037 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:57.954017 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:19:58.258587 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:19:58.258488 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:03.052598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:03.052566 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:03.881672 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:20:03.881636 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:20:07.838476 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:07.838437 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:12.656450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:12.656416 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:17.452329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:17.452294 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:22.252586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:22.252553 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:27.107000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:27.106961 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:29.081393 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:29.081350 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:31.940208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:31.940175 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:33.882479 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:20:33.882436 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:20:35.037789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.037751 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk"] Oct 31 16:20:35.043280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.043259 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.091716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.091689 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e7f13a-openshift-default-dockercfg-m7lj5\"" Oct 31 16:20:35.142610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.142578 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk"] Oct 31 16:20:35.201736 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.201701 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.201899 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.201768 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3460a123-5789-42de-9c62-1b51a177102a-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.201899 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.201880 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.201970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.201943 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.202023 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.202002 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.202063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.202037 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx62m\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-kube-api-access-mx62m\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.202097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.202067 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.202186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.202174 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3460a123-5789-42de-9c62-1b51a177102a-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.202285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.202274 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.303570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303479 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3460a123-5789-42de-9c62-1b51a177102a-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.303749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303572 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.303749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303601 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.303749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303734 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3460a123-5789-42de-9c62-1b51a177102a-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.303921 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303808 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.303921 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303900 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303936 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.303978 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx62m\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-kube-api-access-mx62m\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.304005 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.304082 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304348 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.304328 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.304384 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3460a123-5789-42de-9c62-1b51a177102a-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.304385 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.304511 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.304470 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.306579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.306558 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.306783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.306767 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3460a123-5789-42de-9c62-1b51a177102a-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.356690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.356658 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.356861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.356751 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx62m\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-kube-api-access-mx62m\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:35.657261 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:35.657231 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:36.088735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.088650 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk"] Oct 31 16:20:36.094126 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:20:36.094099 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3460a123_5789_42de_9c62_1b51a177102a.slice/crio-22a031fc19bfeb9e67af6316fc07d1bfe4f46b7b98a2490520d1977896457ab3 WatchSource:0}: Error finding container 22a031fc19bfeb9e67af6316fc07d1bfe4f46b7b98a2490520d1977896457ab3: Status 404 returned error can't find the container with id 22a031fc19bfeb9e67af6316fc07d1bfe4f46b7b98a2490520d1977896457ab3 Oct 31 16:20:36.096686 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.096653 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:20:36.096784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.096716 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:20:36.096784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.096747 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:20:36.516061 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.516027 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" event={"ID":"3460a123-5789-42de-9c62-1b51a177102a","Type":"ContainerStarted","Data":"768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f"} Oct 31 16:20:36.516252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.516111 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" event={"ID":"3460a123-5789-42de-9c62-1b51a177102a","Type":"ContainerStarted","Data":"22a031fc19bfeb9e67af6316fc07d1bfe4f46b7b98a2490520d1977896457ab3"} Oct 31 16:20:36.657730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.657699 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:36.673890 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.673841 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podStartSLOduration=2.673827546 podStartE2EDuration="2.673827546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:20:36.669437638 +0000 UTC m=+1083.704732453" watchObservedRunningTime="2026-10-31 16:20:36.673827546 +0000 UTC m=+1083.709122350" Oct 31 16:20:36.854572 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:36.854468 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:37.195348 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:37.195317 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:37.660181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:37.660124 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 16:20:37.660367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:37.660199 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 16:20:38.658082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:38.658047 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 16:20:38.658485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:38.658098 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 16:20:39.657795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:39.657765 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 16:20:39.657964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:39.657815 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 16:20:40.657667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:40.657627 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 16:20:40.658072 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:40.657677 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 16:20:41.657861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:41.657826 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 16:20:41.658279 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:41.657887 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 16:20:41.777422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:41.777386 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:42.658406 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:42.658370 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 31 16:20:42.658888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:42.658462 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 31 16:20:43.662512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:43.662475 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:43.663285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:43.663259 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:43.664382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:43.664358 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:20:46.555703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:46.555664 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:51.353638 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:51.353578 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:20:56.262085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:20:56.262038 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:01.055270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:01.055234 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:03.883206 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:21:03.883167 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:21:05.797389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:05.797348 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:09.839985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:09.839945 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:10.660364 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:10.660335 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:15.560572 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:15.560482 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:20.363825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.363786 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:20.386271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.386234 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5"] Oct 31 16:21:20.393846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.393818 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482078 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482028 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5cb\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-kube-api-access-vr5cb\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482188 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482231 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482266 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc1fd80-a78f-4942-b879-07baf07ee37a-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482333 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482370 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482402 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482490 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.482830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.482805 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.490528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.490495 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5"] Oct 31 16:21:20.583764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.583722 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc1fd80-a78f-4942-b879-07baf07ee37a-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.583984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.583787 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.583984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.583820 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.583984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.583838 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.583984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.583905 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.583984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.583929 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584035 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5cb\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-kube-api-access-vr5cb\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584087 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584117 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584470 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584323 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-data\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584470 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584364 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584574 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-credential-socket\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584623 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc1fd80-a78f-4942-b879-07baf07ee37a-istiod-ca-cert\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.584725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.584665 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-certs\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.586258 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.586237 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-envoy\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.586569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.586554 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-podinfo\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.636586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.636551 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-token\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.636732 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.636667 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5cb\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-kube-api-access-vr5cb\") pod \"e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:20.715749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:20.715713 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:21.148418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:21.148386 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5"] Oct 31 16:21:21.154312 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:21:21.154281 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc1fd80_a78f_4942_b879_07baf07ee37a.slice/crio-49d9d29f4e94b64a59a2dd5789991c7a115ad9d9b447223380a5737c8fc73930 WatchSource:0}: Error finding container 49d9d29f4e94b64a59a2dd5789991c7a115ad9d9b447223380a5737c8fc73930: Status 404 returned error can't find the container with id 49d9d29f4e94b64a59a2dd5789991c7a115ad9d9b447223380a5737c8fc73930 Oct 31 16:21:21.156860 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:21.156824 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:21:21.156958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:21.156912 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:21:21.156958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:21.156942 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:21:21.890726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:21.890686 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" event={"ID":"ffc1fd80-a78f-4942-b879-07baf07ee37a","Type":"ContainerStarted","Data":"7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275"} Oct 31 16:21:21.891269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:21.890776 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" event={"ID":"ffc1fd80-a78f-4942-b879-07baf07ee37a","Type":"ContainerStarted","Data":"49d9d29f4e94b64a59a2dd5789991c7a115ad9d9b447223380a5737c8fc73930"} Oct 31 16:21:22.095158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:22.095084 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podStartSLOduration=2.095069854 podStartE2EDuration="2.095069854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:22.047630886 +0000 UTC m=+1129.082925692" watchObservedRunningTime="2026-10-31 16:21:22.095069854 +0000 UTC m=+1129.130364676" Oct 31 16:21:22.716811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:22.716783 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:22.718812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:22.718783 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 16:21:22.718957 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:22.718849 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 16:21:23.003483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:23.003394 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:23.716249 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:23.716216 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 16:21:23.716425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:23.716281 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 16:21:24.716234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:24.716199 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 16:21:24.716612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:24.716267 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 16:21:25.400003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:25.399971 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:25.716191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:25.716156 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 16:21:25.716363 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:25.716208 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 16:21:26.449680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:26.449637 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:26.716730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:26.716637 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 16:21:26.716730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:26.716711 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 16:21:27.716515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:27.716468 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 16:21:27.716810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:27.716526 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 16:21:28.721367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:28.721337 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:28.721751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:28.721592 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:28.722127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:28.722111 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:21:29.429427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:29.429398 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:30.238579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:30.238545 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:32.244200 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.244169 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh"] Oct 31 16:21:32.248740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.248712 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.302479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.302436 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh"] Oct 31 16:21:32.307689 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.307660 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:32.342790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.342764 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1685e\"/\"distinct-2-w6-a1685e-kserve-self-signed-certs\"" Oct 31 16:21:32.349512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.349478 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqb5\" (UniqueName: \"kubernetes.io/projected/7c791e49-425c-4b56-8db8-3536bafe22ac-kube-api-access-hvqb5\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.349641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.349597 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-tmp-dir\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.349641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.349634 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-home\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.349730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.349716 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-dshm\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.349770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.349735 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-model-cache\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.349879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.349860 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c791e49-425c-4b56-8db8-3536bafe22ac-tls-certs\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451371 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-tmp-dir\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451605 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451437 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-home\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451605 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451542 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-dshm\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451605 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451566 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-model-cache\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451713 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c791e49-425c-4b56-8db8-3536bafe22ac-tls-certs\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451765 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqb5\" (UniqueName: \"kubernetes.io/projected/7c791e49-425c-4b56-8db8-3536bafe22ac-kube-api-access-hvqb5\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451833 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-tmp-dir\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.451914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.451851 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-home\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.452149 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.452112 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-model-cache\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.454436 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.454413 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-dshm\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.454561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.454423 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c791e49-425c-4b56-8db8-3536bafe22ac-tls-certs\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.504926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.504850 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqb5\" (UniqueName: \"kubernetes.io/projected/7c791e49-425c-4b56-8db8-3536bafe22ac-kube-api-access-hvqb5\") pod \"distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.561863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.561827 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:32.962779 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:32.962746 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh"] Oct 31 16:21:32.967694 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:21:32.967658 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c791e49_425c_4b56_8db8_3536bafe22ac.slice/crio-53d4ecc56ff03f167c635a94b062c6d37e7783ee417c27b930eb2429d1f9f9bf WatchSource:0}: Error finding container 53d4ecc56ff03f167c635a94b062c6d37e7783ee417c27b930eb2429d1f9f9bf: Status 404 returned error can't find the container with id 53d4ecc56ff03f167c635a94b062c6d37e7783ee417c27b930eb2429d1f9f9bf Oct 31 16:21:33.027042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:33.027015 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" event={"ID":"7c791e49-425c-4b56-8db8-3536bafe22ac","Type":"ContainerStarted","Data":"53d4ecc56ff03f167c635a94b062c6d37e7783ee417c27b930eb2429d1f9f9bf"} Oct 31 16:21:33.882521 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:21:33.882484 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:21:34.036292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:34.036257 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" event={"ID":"7c791e49-425c-4b56-8db8-3536bafe22ac","Type":"ContainerStarted","Data":"e9adf2155eab8d9e7feab4cdff98a34b3fa4a46eca3cd721b6da5916f5201109"} Oct 31 16:21:34.144546 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:34.144488 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podStartSLOduration=2.144471523 podStartE2EDuration="2.144471523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:34.141652871 +0000 UTC m=+1141.176947666" watchObservedRunningTime="2026-10-31 16:21:34.144471523 +0000 UTC m=+1141.179766325" Oct 31 16:21:34.562864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:34.562774 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:35.074166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.074101 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:35.498997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.498965 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps"] Oct 31 16:21:35.503438 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.503419 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.600694 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.600659 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.600885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.600717 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.600885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.600825 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwlkz\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-kube-api-access-lwlkz\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.600974 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.600961 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.601043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.601025 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.601101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.601075 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.601174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.601150 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.601344 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.601325 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.601408 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.601380 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.690476 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.690439 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps"] Oct 31 16:21:35.702639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.702605 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.702813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.702662 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.702813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.702702 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwlkz\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-kube-api-access-lwlkz\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.702813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.702803 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.702979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.702829 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.702979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.702864 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.702979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.702895 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.703154 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.703066 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.703154 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.703122 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.703554 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.703529 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.703650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.703533 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.703650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.703587 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.703773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.703744 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.703880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.703863 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.705718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.705698 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.706381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.706361 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.755774 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.755696 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.755914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.755792 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwlkz\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-kube-api-access-lwlkz\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:35.818476 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:35.818447 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:36.250962 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:36.250928 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps"] Oct 31 16:21:36.255985 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:21:36.255960 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2e1add_2456_4d28_a7cf_d1da326a6cc3.slice/crio-fa95619d27c54c4c4fefc2ee9229bb0b1dc2b547a35ad495b29448a3278c9b9a WatchSource:0}: Error finding container fa95619d27c54c4c4fefc2ee9229bb0b1dc2b547a35ad495b29448a3278c9b9a: Status 404 returned error can't find the container with id fa95619d27c54c4c4fefc2ee9229bb0b1dc2b547a35ad495b29448a3278c9b9a Oct 31 16:21:36.258432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:36.258399 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:21:36.258529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:36.258469 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:21:36.258529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:36.258496 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:21:37.058308 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:37.058271 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" event={"ID":"9e2e1add-2456-4d28-a7cf-d1da326a6cc3","Type":"ContainerStarted","Data":"ae81d5e40bd17edb02e911a30ae78588c47874d758d1bd5364d0ced9a7a37f69"} Oct 31 16:21:37.058524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:37.058327 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" event={"ID":"9e2e1add-2456-4d28-a7cf-d1da326a6cc3","Type":"ContainerStarted","Data":"fa95619d27c54c4c4fefc2ee9229bb0b1dc2b547a35ad495b29448a3278c9b9a"} Oct 31 16:21:37.214726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:37.214647 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podStartSLOduration=2.214627051 podStartE2EDuration="2.214627051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:21:37.210593715 +0000 UTC m=+1144.245888531" watchObservedRunningTime="2026-10-31 16:21:37.214627051 +0000 UTC m=+1144.249921861" Oct 31 16:21:37.819233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:37.819197 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:37.839405 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:37.839358 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 16:21:37.839701 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:37.839651 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 16:21:38.819442 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:38.819409 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 16:21:38.819821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:38.819460 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 16:21:39.819217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:39.819168 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 16:21:39.819446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:39.819242 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 16:21:40.006845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:40.006814 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:40.168776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:40.168583 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:40.827402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:40.827366 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 16:21:40.827710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:40.827683 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 16:21:41.819855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:41.819822 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 16:21:41.820426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:41.820397 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 16:21:42.396942 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.396888 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.551161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.543086 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.560163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.558233 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.568168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.565213 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:42.653234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.652237 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.706884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.706841 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.731957 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.731911 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.784283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.733937 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.819650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.819613 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 16:21:42.819898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.819868 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 16:21:42.889522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.889481 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.916222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.911843 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:42.986295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:42.986248 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:43.169253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.138388 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:21:43.305856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.305820 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:21:43.306100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.306070 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:43.437883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.437706 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:21:43.440601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.440573 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:21:43.553757 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.548297 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:43.553757 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.549085 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:21:43.663930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.580279 4880 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:43.752733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.655042 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:43.754009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.753984 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:21:43.773432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.773405 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:21:43.809179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.806390 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:21:43.809179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.808247 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:21:43.809179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.808671 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:21:43.821999 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.821222 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 16:21:43.821999 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.821274 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 16:21:43.831196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.828370 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:21:43.831483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:43.831446 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:21:45.257962 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:45.257926 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:45.258513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:45.258493 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:45.259485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:45.259465 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:21:45.284434 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:45.284396 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:50.082488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:50.082438 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:52.573871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:52.573839 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:52.585045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:52.585019 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:21:55.041029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:55.040993 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:21:59.854220 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:21:59.854186 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:03.882715 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:22:03.882677 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:22:04.028342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.028307 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:04.527428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.527393 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb"] Oct 31 16:22:04.531879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.531850 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.607889 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.607848 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.607971 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.608178 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxmx\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-kube-api-access-xwxmx\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.608302 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.608372 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608460 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.608436 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.608554 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608697 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.608631 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.608697 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.608677 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.662993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.662959 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:04.709522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709491 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxmx\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-kube-api-access-xwxmx\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709709 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709560 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709709 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709614 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709709 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709636 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709711 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709744 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709785 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709807 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.709883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.709876 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.710409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.710049 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.710409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.710345 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.710409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.710383 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.711559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.710428 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.711559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.711339 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.713084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.713029 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.713448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.713405 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.716844 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.716821 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb"] Oct 31 16:22:04.762676 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.762649 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.762936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.762912 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxmx\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-kube-api-access-xwxmx\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.846465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.846388 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:04.937589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.937377 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps"] Oct 31 16:22:04.944782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:04.944709 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.000356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.000309 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps"] Oct 31 16:22:05.012897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.012856 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.012953 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.012997 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.013175 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhqz\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-kube-api-access-dnhqz\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.013228 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.013316 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013407 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.013353 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013519 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.013505 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.013577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.013537 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.114375 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114273 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.114375 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114318 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.114611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114486 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.114611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114533 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.114611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114591 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.114823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114614 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.114823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114654 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.115252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.114979 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.115252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.115237 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.115415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.115056 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.115415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.115125 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.115415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.115034 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.115415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.115233 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhqz\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-kube-api-access-dnhqz\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.115415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.115343 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.117371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.117349 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.117469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.117409 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.168095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.168062 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.168279 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.168106 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhqz\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-kube-api-access-dnhqz\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.259191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.259124 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:05.285921 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.285892 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb"] Oct 31 16:22:05.290629 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:22:05.290593 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2a16d2_824c_460f_bf6d_0fba9cbb47b5.slice/crio-6b53c2e120a42449a292649f10531c65cc9a8514b69a32667f89942850866ea7 WatchSource:0}: Error finding container 6b53c2e120a42449a292649f10531c65cc9a8514b69a32667f89942850866ea7: Status 404 returned error can't find the container with id 6b53c2e120a42449a292649f10531c65cc9a8514b69a32667f89942850866ea7 Oct 31 16:22:05.293230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.293198 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:05.293315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.293261 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:05.293315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.293290 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:05.400599 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.400559 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" event={"ID":"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5","Type":"ContainerStarted","Data":"7f8002e6c17fe3dd4993f9423b6da933e0dd99bc6365632b3bacbd4406395491"} Oct 31 16:22:05.400779 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.400631 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" event={"ID":"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5","Type":"ContainerStarted","Data":"6b53c2e120a42449a292649f10531c65cc9a8514b69a32667f89942850866ea7"} Oct 31 16:22:05.510252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.510196 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podStartSLOduration=1.510182673 podStartE2EDuration="1.510182673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:05.507973686 +0000 UTC m=+1172.543268489" watchObservedRunningTime="2026-10-31 16:22:05.510182673 +0000 UTC m=+1172.545477478" Oct 31 16:22:05.641742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.641700 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps"] Oct 31 16:22:05.646600 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:22:05.646571 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0588aa5a_9e9d_4d41_950a_ad8a4d8fed1d.slice/crio-79398cbe4181b5259d89a6c1e2f3f434ff7e357c7e6d609c27e0b9a9b43ddba3 WatchSource:0}: Error finding container 79398cbe4181b5259d89a6c1e2f3f434ff7e357c7e6d609c27e0b9a9b43ddba3: Status 404 returned error can't find the container with id 79398cbe4181b5259d89a6c1e2f3f434ff7e357c7e6d609c27e0b9a9b43ddba3 Oct 31 16:22:05.649433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.649393 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:05.649505 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.649489 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:05.649560 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.649532 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:05.721272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.721048 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv"] Oct 31 16:22:05.726752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.726731 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.804439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.804405 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv"] Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825338 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825465 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825497 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825625 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825652 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825679 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjq8\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-kube-api-access-zwjq8\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825736 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825813 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.830189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.825837 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.847033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.847006 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:05.927008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.926925 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.926969 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927052 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927078 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927102 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjq8\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-kube-api-access-zwjq8\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927187 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927270 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927301 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927453 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927598 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-certs\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927767 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927722 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-credential-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927860 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927835 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istiod-ca-cert\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927859 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-data\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.927915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.927904 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-socket\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.929780 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.929758 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-envoy\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.929896 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.929879 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-podinfo\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.980721 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.980689 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-token\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:05.981204 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:05.981185 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjq8\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-kube-api-access-zwjq8\") pod \"e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:06.052874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.052844 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:06.408917 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.408879 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" event={"ID":"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d","Type":"ContainerStarted","Data":"8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c"} Oct 31 16:22:06.409097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.408942 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" event={"ID":"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d","Type":"ContainerStarted","Data":"79398cbe4181b5259d89a6c1e2f3f434ff7e357c7e6d609c27e0b9a9b43ddba3"} Oct 31 16:22:06.506926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.506893 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv"] Oct 31 16:22:06.512485 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:22:06.512462 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28e7e01_7194_4569_90ef_311b5d3b5fa2.slice/crio-eb86d27d0afe6f6b6c8db3f05eee4ec6615bab2cc7d266d1323f91567a1d22fc WatchSource:0}: Error finding container eb86d27d0afe6f6b6c8db3f05eee4ec6615bab2cc7d266d1323f91567a1d22fc: Status 404 returned error can't find the container with id eb86d27d0afe6f6b6c8db3f05eee4ec6615bab2cc7d266d1323f91567a1d22fc Oct 31 16:22:06.515511 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.515485 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:06.515575 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.515542 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:06.515575 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.515571 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:06.519943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.519906 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podStartSLOduration=2.519893167 podStartE2EDuration="2.519893167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:06.516534176 +0000 UTC m=+1173.551828974" watchObservedRunningTime="2026-10-31 16:22:06.519893167 +0000 UTC m=+1173.555187972" Oct 31 16:22:06.877458 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.877364 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 16:22:06.877690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:06.877667 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 16:22:07.266161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.259508 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:07.277173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.276313 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 16:22:07.277173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.276376 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 16:22:07.418036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.418003 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" event={"ID":"a28e7e01-7194-4569-90ef-311b5d3b5fa2","Type":"ContainerStarted","Data":"3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1"} Oct 31 16:22:07.418188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.418059 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" event={"ID":"a28e7e01-7194-4569-90ef-311b5d3b5fa2","Type":"ContainerStarted","Data":"eb86d27d0afe6f6b6c8db3f05eee4ec6615bab2cc7d266d1323f91567a1d22fc"} Oct 31 16:22:07.540210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.535631 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podStartSLOduration=2.53561032 podStartE2EDuration="2.53561032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:07.533569048 +0000 UTC m=+1174.568863851" watchObservedRunningTime="2026-10-31 16:22:07.53561032 +0000 UTC m=+1174.570905124" Oct 31 16:22:07.856609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.856512 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 16:22:07.856873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:07.856842 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 16:22:08.069158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:08.068921 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:08.069158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:08.069094 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 16:22:08.069409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:08.069169 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 16:22:08.266162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:08.265481 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 16:22:08.266162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:08.265542 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 16:22:08.855160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:08.854545 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 16:22:08.855160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:08.854611 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 16:22:09.061558 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:09.061521 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 16:22:09.061877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:09.061835 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 16:22:09.263161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:09.259563 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 16:22:09.263161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:09.259624 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 16:22:09.848160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:09.847598 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 16:22:09.848160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:09.847659 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 16:22:09.940159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:09.939660 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:10.060162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:10.057548 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 16:22:10.060162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:10.057608 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 16:22:10.273158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:10.265574 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 16:22:10.273158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:10.265631 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 16:22:10.919200 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:10.916202 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 16:22:10.919200 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:10.916257 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 16:22:11.108164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:11.096513 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 16:22:11.108164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:11.096574 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 16:22:11.298362 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:11.298267 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 16:22:11.298857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:11.298826 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 16:22:11.852531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:11.852491 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 16:22:11.853487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:11.853456 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 16:22:12.060574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:12.060537 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 16:22:12.061191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:12.061159 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 16:22:12.265158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:12.264775 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 16:22:12.265158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:12.264839 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 16:22:12.847682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:12.847647 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 16:22:12.847980 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:12.847934 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 16:22:13.054659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:13.054623 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 16:22:13.054903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:13.054876 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 16:22:13.260514 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:13.260472 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 16:22:13.261005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:13.260544 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 16:22:13.852842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:13.852811 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:13.853284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:13.853239 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:13.857454 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:13.857435 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:22:14.053791 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:14.053751 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 16:22:14.053959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:14.053832 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 16:22:14.265908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:14.265873 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:14.266399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:14.266374 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:14.267103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:14.267078 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:22:15.242677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:15.242637 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:16.140380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:16.140344 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:22:16.140988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:16.140960 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:17.189579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:17.189544 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:22:17.190175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:17.190126 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:17.392450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:17.392415 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:17.394177 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:17.394157 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:17.398355 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:17.398334 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:22:19.932068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:19.932034 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x"] Oct 31 16:22:19.936420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:19.936399 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:19.989854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:19.989823 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x"] Oct 31 16:22:20.103373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103340 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103372 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a0d474-ca84-45b8-8711-a944b019383c-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103462 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a0d474-ca84-45b8-8711-a944b019383c-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103498 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pvl\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-kube-api-access-z9pvl\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103569 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103613 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103679 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103696 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.103834 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.103823 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.204760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.204684 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a0d474-ca84-45b8-8711-a944b019383c-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.204760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.204720 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pvl\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-kube-api-access-z9pvl\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.204760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.204741 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.204772 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.204823 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.204840 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205031 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205202 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205228 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a0d474-ca84-45b8-8711-a944b019383c-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205234 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-data\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205262 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-credential-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205501 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-socket\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205570 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-certs\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.205727 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.205698 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a0d474-ca84-45b8-8711-a944b019383c-istiod-ca-cert\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.207335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.207315 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-envoy\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.207604 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.207587 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a0d474-ca84-45b8-8711-a944b019383c-istio-podinfo\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.257886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.257856 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-istio-token\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.258002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.257926 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pvl\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-kube-api-access-z9pvl\") pod \"e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:20.443880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.443840 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:20.551833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:20.551747 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:21.047969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:21.047947 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x"] Oct 31 16:22:21.049624 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:22:21.049597 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a0d474_ca84_45b8_8711_a944b019383c.slice/crio-622e6b1aa2f40c3e51d1aeb25eba6bc5a3e9d8291903d4d101dfa9f3807d246b WatchSource:0}: Error finding container 622e6b1aa2f40c3e51d1aeb25eba6bc5a3e9d8291903d4d101dfa9f3807d246b: Status 404 returned error can't find the container with id 622e6b1aa2f40c3e51d1aeb25eba6bc5a3e9d8291903d4d101dfa9f3807d246b Oct 31 16:22:21.052047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:21.052018 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:21.052130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:21.052089 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:21.052130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:21.052120 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:21.890627 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:21.890586 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" event={"ID":"15a0d474-ca84-45b8-8711-a944b019383c","Type":"ContainerStarted","Data":"9ebbc37f384c3b648e30d1e9d3ba66e2afabf9b8e2f75ab127477a5c6e6ea4c0"} Oct 31 16:22:21.890800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:21.890647 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" event={"ID":"15a0d474-ca84-45b8-8711-a944b019383c","Type":"ContainerStarted","Data":"622e6b1aa2f40c3e51d1aeb25eba6bc5a3e9d8291903d4d101dfa9f3807d246b"} Oct 31 16:22:22.552925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:22.552889 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:22.554770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:22.554743 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 16:22:22.554901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:22.554800 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 16:22:23.552616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:23.552576 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 16:22:23.552795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:23.552644 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 16:22:24.552280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:24.552240 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 16:22:24.552663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:24.552296 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 16:22:25.391791 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:25.391755 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:25.552760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:25.552629 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 16:22:25.552760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:25.552694 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 16:22:26.552826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:26.552790 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 16:22:26.553220 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:26.552862 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 16:22:27.552359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:27.552324 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 16:22:27.552562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:27.552386 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 16:22:28.557278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:28.557250 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:28.557781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:28.557759 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:28.558339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:28.558318 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:22:28.712114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:28.712053 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podStartSLOduration=9.71203545 podStartE2EDuration="9.71203545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:22.049883223 +0000 UTC m=+1189.085178027" watchObservedRunningTime="2026-10-31 16:22:28.71203545 +0000 UTC m=+1195.747330255" Oct 31 16:22:29.751938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:29.751908 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:30.256115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:30.256072 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:33.251056 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:33.251022 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:22:33.251566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:33.251469 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:22:33.259253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:33.259229 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:22:33.259438 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:33.259372 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:22:33.274452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:33.274431 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:22:33.274452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:33.274448 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:22:33.887065 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:22:33.887028 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:22:34.937185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:34.937150 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24"] Oct 31 16:22:34.942738 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:34.942716 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.026390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.026352 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.026588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.026565 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.026730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.026713 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.026789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.026772 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rzsd\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-kube-api-access-5rzsd\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.026826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.026812 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6136a561-9fbf-4010-bfe4-40feb1408578-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.026943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.026923 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.027008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.026959 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.027152 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.027119 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6136a561-9fbf-4010-bfe4-40feb1408578-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.027246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.027230 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.044857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.044828 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24"] Oct 31 16:22:35.128348 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128317 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rzsd\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-kube-api-access-5rzsd\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128360 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6136a561-9fbf-4010-bfe4-40feb1408578-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128409 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128431 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128536 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6136a561-9fbf-4010-bfe4-40feb1408578-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128620 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128682 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128848 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128833 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128886 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.128956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128917 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.129008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.128911 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-certs\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.129167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.129124 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6136a561-9fbf-4010-bfe4-40feb1408578-istiod-ca-cert\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.129257 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.129234 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-credential-socket\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.129298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.129239 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-data\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.130866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.130834 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-envoy\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.130995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.130974 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6136a561-9fbf-4010-bfe4-40feb1408578-istio-podinfo\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.185975 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.185943 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-istio-token\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.186101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.186072 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rzsd\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-kube-api-access-5rzsd\") pod \"e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.257404 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.257315 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:35.259884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.259851 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:35.703179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.703087 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24"] Oct 31 16:22:35.707263 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:22:35.707236 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6136a561_9fbf_4010_bfe4_40feb1408578.slice/crio-d984dff26cf90506e88d7b78628efa77127f5afb5ac88dc0a8582a7c77bc5539 WatchSource:0}: Error finding container d984dff26cf90506e88d7b78628efa77127f5afb5ac88dc0a8582a7c77bc5539: Status 404 returned error can't find the container with id d984dff26cf90506e88d7b78628efa77127f5afb5ac88dc0a8582a7c77bc5539 Oct 31 16:22:35.710253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.710225 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:35.710323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.710287 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:35.710323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.710319 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:35.930817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.930788 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84"] Oct 31 16:22:35.935247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.935225 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:35.957786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.957720 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct"] Oct 31 16:22:35.962052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.962030 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:35.991727 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:35.991699 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84"] Oct 31 16:22:36.035352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.035312 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" event={"ID":"6136a561-9fbf-4010-bfe4-40feb1408578","Type":"ContainerStarted","Data":"a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4"} Oct 31 16:22:36.035530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.035379 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" event={"ID":"6136a561-9fbf-4010-bfe4-40feb1408578","Type":"ContainerStarted","Data":"d984dff26cf90506e88d7b78628efa77127f5afb5ac88dc0a8582a7c77bc5539"} Oct 31 16:22:36.046296 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.046257 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct"] Oct 31 16:22:36.053835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.053801 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.054076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.054050 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmtl\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-kube-api-access-tgmtl\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.054250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.054232 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.054378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.054361 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.054529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.054512 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.054658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.054641 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.054819 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.054802 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.054960 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.054945 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.055086 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.055071 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.055317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.055297 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p9w9\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-kube-api-access-9p9w9\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.055465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.055445 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.055642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.055623 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.055782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.055765 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.056003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.055986 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.056197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.056180 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.056437 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.056396 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.056880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.056854 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.057283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.057261 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159293 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159333 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159373 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.159497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159405 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.159497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159441 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159497 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p9w9\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-kube-api-access-9p9w9\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.159778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159529 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.159778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159554 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.159778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159597 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.159778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159730 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159774 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159803 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.159984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159839 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159861 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.159984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.159919 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.160276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160031 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.160276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160065 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.160276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160105 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmtl\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-kube-api-access-tgmtl\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.160276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160152 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.160276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160158 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.160276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160224 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.160609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160348 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-data\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.160731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160704 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.160806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160742 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.160875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.160851 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.161100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.161060 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istiod-ca-cert\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.161409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.161386 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-credential-socket\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.161541 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.161520 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-certs\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.162634 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.162581 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.162753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.162680 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.163024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.163004 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-envoy\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.163331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.163308 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-podinfo\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.212376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.212305 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p9w9\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-kube-api-access-9p9w9\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.212924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.212895 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.213352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.213330 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-token\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.213754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.213734 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmtl\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-kube-api-access-tgmtl\") pod \"e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.250849 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.250818 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:36.258212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.258190 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:36.259517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.259499 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:36.259602 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.259538 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:36.279909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.279866 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podStartSLOduration=2.279853924 podStartE2EDuration="2.279853924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:36.275196294 +0000 UTC m=+1203.310491099" watchObservedRunningTime="2026-10-31 16:22:36.279853924 +0000 UTC m=+1203.315148729" Oct 31 16:22:36.284610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.284590 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:36.649438 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.649401 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84"] Oct 31 16:22:36.650432 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:22:36.650406 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9251b2_071b_4d53_bdea_3aa2efd0f90c.slice/crio-2761639283a6ab742d632b634b6e473fc0ee108a6ca1022649fd3dcf8353ef16 WatchSource:0}: Error finding container 2761639283a6ab742d632b634b6e473fc0ee108a6ca1022649fd3dcf8353ef16: Status 404 returned error can't find the container with id 2761639283a6ab742d632b634b6e473fc0ee108a6ca1022649fd3dcf8353ef16 Oct 31 16:22:36.653230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.653190 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:36.653299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.653274 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:36.653336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.653303 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:36.693264 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:22:36.693230 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f9f219_fdf3_42cc_8ac5_e6f4588649c2.slice/crio-a6295638f0dede7b273124ed5e1933310a2701ce0fcda2f225e569a8ae917a2c WatchSource:0}: Error finding container a6295638f0dede7b273124ed5e1933310a2701ce0fcda2f225e569a8ae917a2c: Status 404 returned error can't find the container with id a6295638f0dede7b273124ed5e1933310a2701ce0fcda2f225e569a8ae917a2c Oct 31 16:22:36.695087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.695062 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct"] Oct 31 16:22:36.696835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.696798 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:36.696943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.696907 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:36.697013 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:36.696950 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:22:37.054619 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.054518 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" event={"ID":"3f9251b2-071b-4d53-bdea-3aa2efd0f90c","Type":"ContainerStarted","Data":"1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6"} Oct 31 16:22:37.054619 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.054590 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" event={"ID":"3f9251b2-071b-4d53-bdea-3aa2efd0f90c","Type":"ContainerStarted","Data":"2761639283a6ab742d632b634b6e473fc0ee108a6ca1022649fd3dcf8353ef16"} Oct 31 16:22:37.057919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.057888 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" event={"ID":"95f9f219-fdf3-42cc-8ac5-e6f4588649c2","Type":"ContainerStarted","Data":"fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b"} Oct 31 16:22:37.058048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.057937 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" event={"ID":"95f9f219-fdf3-42cc-8ac5-e6f4588649c2","Type":"ContainerStarted","Data":"a6295638f0dede7b273124ed5e1933310a2701ce0fcda2f225e569a8ae917a2c"} Oct 31 16:22:37.170396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.170344 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podStartSLOduration=2.170328724 podStartE2EDuration="2.170328724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:37.165085483 +0000 UTC m=+1204.200380287" watchObservedRunningTime="2026-10-31 16:22:37.170328724 +0000 UTC m=+1204.205623529" Oct 31 16:22:37.252443 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.252408 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:37.253997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.253965 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:37.254166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.254040 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:37.258425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.258397 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:37.258562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.258447 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:37.276377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.276332 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podStartSLOduration=2.27631985 podStartE2EDuration="2.27631985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:22:37.272307862 +0000 UTC m=+1204.307602666" watchObservedRunningTime="2026-10-31 16:22:37.27631985 +0000 UTC m=+1204.311614653" Oct 31 16:22:37.284849 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.284825 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:37.286588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.286567 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:37.286670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:37.286610 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:38.257178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.251681 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:38.257178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.251737 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:38.258428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.258219 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:38.258428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.258267 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:38.307162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.291493 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:38.307162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.291555 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:38.851163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.840233 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:38.878278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.878228 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 16:22:38.908163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.902229 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:38.994319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:38.994272 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.134903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.134862 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.260824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.260778 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:39.265157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.261388 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:39.265157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.261758 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:39.265157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.261801 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:39.298399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.298363 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:39.303236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.303198 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:39.441339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.441231 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.542000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.541957 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.697380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.697266 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.739097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.738680 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 16:22:39.780795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.780752 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.808271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.808225 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.830267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.830223 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.858161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.850273 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.858161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.851194 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:22:39.887159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.886720 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.887159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.887119 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:22:39.908163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.904221 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:39.908163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.904653 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:22:40.024217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:39.999698 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:40.024217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.000096 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:22:40.038729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.034289 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:22:40.038729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.034372 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:40.236772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.164893 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:40.236772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.165289 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:22:40.345216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.313190 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:40.345216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.313241 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:40.345216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.313309 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:40.345216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.313330 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:40.345216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.327178 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:40.345216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.327229 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:40.603099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.581434 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:40.727163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.702797 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 16:22:40.727163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.705101 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:22:40.760378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.759236 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:40.760715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.760694 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:22:40.823163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.798264 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:22:40.837238 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.837207 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:22:40.855149 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.855027 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 16:22:40.857646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.856211 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:22:40.866218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.856242 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:22:40.881287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.881110 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:22:40.881585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.881564 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:22:40.915813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.915776 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:40.931669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:40.931631 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 16:22:41.091168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.056371 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 16:22:41.264223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.186927 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:41.322160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.313565 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:41.336415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.336374 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:41.336598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.336438 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:41.336598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.336524 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:41.336598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.336546 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:41.345164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.339548 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:41.345164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.339602 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:41.589248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.586434 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:41.589248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.586984 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:22:41.724615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.703492 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:41.728324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.728217 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Oct 31 16:22:41.777161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.765295 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 16:22:41.849353 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.849254 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:41.932244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.884386 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:22:41.932244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.885577 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 16:22:41.932244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.885886 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:22:41.966161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.933740 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:41.966161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.933803 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:41.966161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:41.934455 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.117216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.113620 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 16:22:42.212228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.189231 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.292642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.264889 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:42.292642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.264949 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:42.292642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.290688 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:42.292642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.290758 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:42.294480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.294228 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:42.294480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.294280 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:42.579306 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.579256 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.654320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.630604 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.714317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.714272 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.799683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.779345 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.836215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.810407 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:42.867115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.864705 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:22:42.899340 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.899294 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.904678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.904242 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.916110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.904281 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:22:42.934294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.934245 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:42.934936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:42.934905 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:43.264159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.249224 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 16:22:43.305230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.304252 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:22:43.619226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.601820 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:43.619226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.601894 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:43.635272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.632999 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:43.644848 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.642541 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:43.644848 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.642601 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:43.645104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.644889 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:43.645104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.644934 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:43.647396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.647363 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Oct 31 16:22:43.754163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.727387 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 16:22:43.813159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.812180 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:43.941158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.940260 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:22:43.954091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.946212 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:43.973751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.973710 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:22:44.001252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:43.987203 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:22:44.008163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.003992 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:44.022938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.022908 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:22:44.065129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.065093 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:44.217546 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.217454 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:22:44.263159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.257092 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:44.263159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.261767 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:44.263159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.258786 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:44.263159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.261846 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:44.322335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.322302 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:44.322580 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.322553 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:44.396198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.389006 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:22:44.608195 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.607995 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:22:44.610486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.610445 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:44.611299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.611277 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:22:44.747159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.740548 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:22:44.799166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.781455 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:22:44.815498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.815423 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 16:22:44.841807 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.841780 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:22:44.976157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:44.975366 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:45.041159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.020266 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:45.253225 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.253068 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:45.254824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.253120 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:45.270005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.269974 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:45.270376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.270341 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:45.289954 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.289920 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:45.290423 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.290375 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:45.537227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.512955 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:22:45.806459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.806358 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:45.810263 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.810236 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:22:45.859347 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.859311 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:45.896810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.896779 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:22:46.004161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:45.993819 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:22:46.264625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:46.264587 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 16:22:46.264859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:46.264832 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 16:22:46.265066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:46.265038 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:46.265210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:46.265186 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:46.292943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:46.292907 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:46.293234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:46.293205 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:47.252118 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:47.252077 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 16:22:47.252118 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:47.252162 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 16:22:47.285431 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:47.285389 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 16:22:47.285600 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:47.285458 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 16:22:47.692318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:47.692289 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:47.692893 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:47.692868 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:47.694239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:47.694218 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:22:48.435320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:48.435266 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:48.553042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:48.553001 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:48.554941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:48.554917 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:48.555781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:48.555763 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:22:48.558625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:48.558602 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:48.559880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:48.559862 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:48.560584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:48.560564 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:22:53.258677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:53.258643 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:22:58.190654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:22:58.190562 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:02.967823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:02.967789 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:03.886825 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:23:03.886787 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:23:07.967628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:07.967594 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:09.043635 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.043601 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7198-62dc83"] Oct 31 16:23:09.049633 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.049612 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:09.103366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.103333 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-62dc83"] Oct 31 16:23:09.141725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.141694 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-szvbq\"" Oct 31 16:23:09.161521 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.161491 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v577l\" (UniqueName: \"kubernetes.io/projected/614b69d1-cb60-433c-99f3-be96c2244b12-kube-api-access-v577l\") pod \"test-curl-7198-62dc83\" (UID: \"614b69d1-cb60-433c-99f3-be96c2244b12\") " pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:09.263098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.263058 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v577l\" (UniqueName: \"kubernetes.io/projected/614b69d1-cb60-433c-99f3-be96c2244b12-kube-api-access-v577l\") pod \"test-curl-7198-62dc83\" (UID: \"614b69d1-cb60-433c-99f3-be96c2244b12\") " pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:09.317670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.317602 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v577l\" (UniqueName: \"kubernetes.io/projected/614b69d1-cb60-433c-99f3-be96c2244b12-kube-api-access-v577l\") pod \"test-curl-7198-62dc83\" (UID: \"614b69d1-cb60-433c-99f3-be96c2244b12\") " pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:09.362965 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.362939 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:09.750310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.750278 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-62dc83"] Oct 31 16:23:09.755938 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:23:09.755910 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod614b69d1_cb60_433c_99f3_be96c2244b12.slice/crio-5344d6331c2d73726b8e1fb1aa544e3117ce3acc04202e72a072a0e007cecf69 WatchSource:0}: Error finding container 5344d6331c2d73726b8e1fb1aa544e3117ce3acc04202e72a072a0e007cecf69: Status 404 returned error can't find the container with id 5344d6331c2d73726b8e1fb1aa544e3117ce3acc04202e72a072a0e007cecf69 Oct 31 16:23:09.758128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:09.758112 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:23:10.587455 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:10.587422 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-62dc83" event={"ID":"614b69d1-cb60-433c-99f3-be96c2244b12","Type":"ContainerStarted","Data":"5344d6331c2d73726b8e1fb1aa544e3117ce3acc04202e72a072a0e007cecf69"} Oct 31 16:23:11.596384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:11.596349 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-62dc83" event={"ID":"614b69d1-cb60-433c-99f3-be96c2244b12","Type":"ContainerStarted","Data":"992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c"} Oct 31 16:23:11.700851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:11.700786 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7198-62dc83" podStartSLOduration=1.330416897 podStartE2EDuration="2.700766629s" totalImagesPullingTime="1.370349732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:11.699993735 +0000 UTC m=+1238.735288540" watchObservedRunningTime="2026-10-31 16:23:11.700766629 +0000 UTC m=+1238.736061434" Oct 31 16:23:12.766754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:12.766722 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:13.070100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:13.069928 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7198-62dc83"] Oct 31 16:23:13.074582 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:13.074557 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7198-62dc83"] Oct 31 16:23:13.610765 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:13.610724 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7198-62dc83" podUID="614b69d1-cb60-433c-99f3-be96c2244b12" containerName="test-curl-7198-62dc83" containerID="cri-o://992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c" gracePeriod=2 Oct 31 16:23:15.862870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:15.862838 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:15.978628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:15.978546 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614b69d1-cb60-433c-99f3-be96c2244b12-kube-api-access-v577l\" (UniqueName: \"kubernetes.io/projected/614b69d1-cb60-433c-99f3-be96c2244b12-kube-api-access-v577l\") pod \"614b69d1-cb60-433c-99f3-be96c2244b12\" (UID: \"614b69d1-cb60-433c-99f3-be96c2244b12\") " Oct 31 16:23:16.005451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:16.005418 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614b69d1-cb60-433c-99f3-be96c2244b12-kube-api-access-v577l" pod "614b69d1-cb60-433c-99f3-be96c2244b12" (UID: "614b69d1-cb60-433c-99f3-be96c2244b12"). InnerVolumeSpecName "kube-api-access-v577l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:23:16.086836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:16.081496 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v577l\" (UniqueName: \"kubernetes.io/projected/614b69d1-cb60-433c-99f3-be96c2244b12-kube-api-access-v577l\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:16.683167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:16.680382 4880 generic.go:356] "Generic (PLEG): container finished" podID="614b69d1-cb60-433c-99f3-be96c2244b12" containerID="992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c" exitCode=137 Oct 31 16:23:16.683167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:16.680574 4880 scope.go:122] "RemoveContainer" containerID="992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c" Oct 31 16:23:16.683167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:16.680753 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:17.037232 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:23:17.032445 4880 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-62dc83\" is forbidden: User \"system:node:ip-10-0-142-127.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-127.us-west-2.compute.internal' and this object" podUID="614b69d1-cb60-433c-99f3-be96c2244b12" pod="opendatahub/test-curl-7198-62dc83" Oct 31 16:23:17.120159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.118338 4880 scope.go:122] "RemoveContainer" containerID="992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c" Oct 31 16:23:17.126158 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:23:17.124280 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c\": container with ID starting with 992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c not found: ID does not exist" containerID="992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c" Oct 31 16:23:17.126158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.124338 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c"} err="failed to get container status \"992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c\": rpc error: code = NotFound desc = could not find container \"992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c\": container with ID starting with 992a3c625686a0163d9d9b1f7bf7b04a3aec411a9a02b70d7d043473d839305c not found: ID does not exist" Oct 31 16:23:17.126158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.125458 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:17.220844 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.220799 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:17.241361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.241326 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614b69d1-cb60-433c-99f3-be96c2244b12" path="/var/lib/kubelet/pods/614b69d1-cb60-433c-99f3-be96c2244b12/volumes" Oct 31 16:23:17.513295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.513244 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:17.643545 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.643492 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:17.793171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.791307 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 16:23:17.793171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.791668 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 16:23:17.793171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.791932 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 31 16:23:17.932159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.927695 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:23:17.990911 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.932491 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:17.990911 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.932521 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:17.990911 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.932544 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 16:23:17.990911 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:17.972228 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:18.137607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.137560 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 16:23:18.143159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.139921 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:23:18.261158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.258473 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:23:18.261158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.258801 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:23:18.463220 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.453230 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:18.644175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.631447 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:23:18.644175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.632913 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:23:18.694495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.665271 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:18.698082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.698042 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:23:18.807216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.794237 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:18.807626 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.807598 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:23:18.992374 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.992327 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:18.993009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.992980 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:18.993400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.993377 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 16:23:18.993691 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.993669 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 16:23:18.994249 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.994231 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:23:19.005239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:18.994406 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:23:19.319186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.005502 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:23:19.344460 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.152091 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 16:23:19.345260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.266304 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:19.463278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.463230 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:19.555159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.548028 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:19.724237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.634399 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:19.743159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.730958 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 16:23:19.789202 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.781643 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 16:23:19.814565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.810561 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:19.851163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.847819 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:23:19.868158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.864440 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:23:19.868158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:19.864535 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:23:20.152232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.060811 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.266161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.065212 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.266161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.065250 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.266161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.065270 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.403215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.361419 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:23:20.492307 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.364596 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.492307 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.364730 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:23:20.706219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.703198 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.755163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.747353 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.846163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.795848 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:20.846163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.797036 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:23:20.953177 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:20.824284 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.175166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.166919 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:21.255162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.255021 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.280162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.255353 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 16:23:21.280162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.255548 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 16:23:21.280162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.255735 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:23:21.435462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.435353 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 16:23:21.463166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.452545 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.513966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.475309 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.513966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.500472 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:23:21.742302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.731947 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.742302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.732056 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:23:21.764615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.757091 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.824186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.819097 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:23:21.824186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.820771 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.903585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.903234 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.914163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.910129 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:23:21.919763 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.919697 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:23:21.942754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.942730 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:23:21.956708 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.955639 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:21.968563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.968540 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:23:21.978569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.976502 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:23:21.992446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.992071 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:23:21.997517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:21.997303 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:23:22.002873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:22.002840 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:23:22.019330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:22.008283 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:23:22.019330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:22.017079 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:23:26.154232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:26.154176 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:30.088799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:30.088758 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:30.932677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:30.932644 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:31.523763 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:31.523725 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:31.684971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:31.684933 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:32.347273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:32.347237 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:33.882057 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:23:33.882023 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:23:34.974185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:34.974118 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht"] Oct 31 16:23:34.974791 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:34.974775 4880 cpu_manager.go:414] "containerMap: removing container" podUID="614b69d1-cb60-433c-99f3-be96c2244b12" containerName="test-curl-7198-62dc83" Oct 31 16:23:34.974832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:34.974792 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614b69d1-cb60-433c-99f3-be96c2244b12" containerName="test-curl-7198-62dc83" Oct 31 16:23:34.974928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:34.974919 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="614b69d1-cb60-433c-99f3-be96c2244b12" containerName="test-curl-7198-62dc83" Oct 31 16:23:34.980291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:34.980269 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.081769 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.081730 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht"] Oct 31 16:23:35.109680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.108989 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.109979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.109953 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.110165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.110129 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.110388 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.110370 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.110512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.110499 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.110608 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.110596 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.110723 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.110707 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.111008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.110983 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.111888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.111861 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r685z\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-kube-api-access-r685z\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213003 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r685z\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-kube-api-access-r685z\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213224 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213193 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213289 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213357 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213342 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213379 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213416 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213441 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213469 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213612 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.213662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.213620 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-certs\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.214097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.214069 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istiod-ca-cert\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.214292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.214207 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.214413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.214360 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-credential-socket\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.214413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.214403 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-data\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.215645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.215625 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-envoy\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.216384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.216363 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-podinfo\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.266569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.266497 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-token\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.266708 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.266605 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r685z\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-kube-api-access-r685z\") pod \"e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.294940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.294906 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:35.722292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.722259 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht"] Oct 31 16:23:35.727289 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:23:35.727256 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ca0f4d_299b_4495_a6aa_73a6e3aedae3.slice/crio-f0b835f4b4d2d52dd2361619194e623983cb73a57542c6cea317b6a6ae2f208e WatchSource:0}: Error finding container f0b835f4b4d2d52dd2361619194e623983cb73a57542c6cea317b6a6ae2f208e: Status 404 returned error can't find the container with id f0b835f4b4d2d52dd2361619194e623983cb73a57542c6cea317b6a6ae2f208e Oct 31 16:23:35.729790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.729757 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:23:35.729866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.729824 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:23:35.729866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.729853 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:23:35.759951 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:35.759923 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:36.036264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.036169 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" event={"ID":"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3","Type":"ContainerStarted","Data":"21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f"} Oct 31 16:23:36.036264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.036227 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" event={"ID":"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3","Type":"ContainerStarted","Data":"f0b835f4b4d2d52dd2361619194e623983cb73a57542c6cea317b6a6ae2f208e"} Oct 31 16:23:36.146357 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.146306 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podStartSLOduration=2.146293443 podStartE2EDuration="2.146293443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:36.143760719 +0000 UTC m=+1263.179055523" watchObservedRunningTime="2026-10-31 16:23:36.146293443 +0000 UTC m=+1263.181588258" Oct 31 16:23:36.299614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.299278 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:36.301120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.301090 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 16:23:36.301262 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.301160 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 16:23:36.397168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.397116 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw"] Oct 31 16:23:36.397365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.397331 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" containerID="cri-o://cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0" gracePeriod=30 Oct 31 16:23:36.741244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.729290 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:36.880323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:36.880288 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:37.047160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:37.046646 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:37.296039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:37.296006 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 16:23:37.296205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:37.296061 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 16:23:38.296171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:38.296117 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 16:23:38.296699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:38.296192 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 16:23:38.496257 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:38.496223 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:38.665130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:38.665093 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:38.810368 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:38.810318 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:38.963896 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:38.963798 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:39.296056 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:39.295941 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 16:23:39.296056 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:39.296034 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 16:23:40.300305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:40.300266 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 16:23:40.300805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:40.300327 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 16:23:40.301819 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:40.301787 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:40.476925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:40.476890 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:41.074854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.074830 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:41.296251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.295946 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 16:23:41.296251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.295997 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 16:23:41.743943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.743917 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:23:41.793391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.793357 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-podinfo\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.793559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.793410 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-envoy\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.793559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.793441 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-token\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-token\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.793559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.793476 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-data\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.793882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.793856 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-data" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:23:41.794172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.794034 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-credential-socket\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.794172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.794074 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-socket\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.794655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.794329 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-credential-socket" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:23:41.794655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.794400 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-socket" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:23:41.794655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.794410 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-kube-api-access-657dt\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-kube-api-access-657dt\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.794655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.794573 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istiod-ca-cert\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.794882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.794657 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-certs\") pod \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\" (UID: \"3d4c9404-b8ae-4922-8e86-1d90533df8d9\") " Oct 31 16:23:41.795374 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.795348 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istiod-ca-cert" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:23:41.795488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.795412 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.795488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.795432 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.795488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.795448 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.795778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.795737 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-certs" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:23:41.796168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.796118 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-podinfo" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:23:41.796302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.796192 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-envoy" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:23:41.796597 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.796576 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-kube-api-access-657dt" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "kube-api-access-657dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:23:41.796996 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.796977 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-token" pod "3d4c9404-b8ae-4922-8e86-1d90533df8d9" (UID: "3d4c9404-b8ae-4922-8e86-1d90533df8d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:23:41.897355 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.897318 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.897355 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.897358 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.897773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.897375 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.897773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.897388 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.897773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.897402 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:41.897773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:41.897414 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657dt\" (UniqueName: \"kubernetes.io/projected/3d4c9404-b8ae-4922-8e86-1d90533df8d9-kube-api-access-657dt\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:23:42.100908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.100811 4880 generic.go:356] "Generic (PLEG): container finished" podID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerID="cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0" exitCode=0 Oct 31 16:23:42.100908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.100860 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" Oct 31 16:23:42.100908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.100895 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" event={"ID":"3d4c9404-b8ae-4922-8e86-1d90533df8d9","Type":"ContainerDied","Data":"cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0"} Oct 31 16:23:42.101197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.100949 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw" event={"ID":"3d4c9404-b8ae-4922-8e86-1d90533df8d9","Type":"ContainerDied","Data":"1abc4713bdfda510a47908de0c5727724a818cb04dd6701bad61b9682d8245db"} Oct 31 16:23:42.101197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.100979 4880 scope.go:122] "RemoveContainer" containerID="cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0" Oct 31 16:23:42.114189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.114172 4880 scope.go:122] "RemoveContainer" containerID="cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0" Oct 31 16:23:42.114534 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:23:42.114513 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0\": container with ID starting with cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0 not found: ID does not exist" containerID="cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0" Oct 31 16:23:42.114594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.114543 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0"} err="failed to get container status \"cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0\": rpc error: code = NotFound desc = could not find container \"cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0\": container with ID starting with cde07d8a4d73e9657ddee0a5c2fed73d0f97b9d5508e14f0f7192c6e2f0c67e0 not found: ID does not exist" Oct 31 16:23:42.264557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.264524 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw"] Oct 31 16:23:42.268472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.268448 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-vkxlw"] Oct 31 16:23:42.296502 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.296470 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 16:23:42.296666 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:42.296524 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 16:23:43.093249 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:43.093208 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" path="/var/lib/kubelet/pods/3d4c9404-b8ae-4922-8e86-1d90533df8d9/volumes" Oct 31 16:23:43.299387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:43.299361 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:43.299851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:43.299832 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:43.300475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:43.300447 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:23:43.518822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:43.518793 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:43.706954 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:43.706924 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:44.091285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:44.091258 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:44.737201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:44.737168 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:44.917083 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:44.917043 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:45.062998 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:45.062916 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:45.220461 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:45.220422 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:45.384114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:45.384079 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:46.445492 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:46.445457 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:46.623532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:46.623496 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:46.784994 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:46.784916 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:48.935463 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:48.935422 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:49.115645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.115605 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:49.276881 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.276715 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:49.389970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.389941 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws"] Oct 31 16:23:49.390614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.390597 4880 cpu_manager.go:414] "containerMap: removing container" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" Oct 31 16:23:49.390614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.390614 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" Oct 31 16:23:49.390808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.390793 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4c9404-b8ae-4922-8e86-1d90533df8d9" containerName="istio-proxy" Oct 31 16:23:49.395931 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.395913 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.464162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.453481 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:49.464162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.455101 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws"] Oct 31 16:23:49.487090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.487062 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a08562e5-dockercfg-w22xf\"" Oct 31 16:23:49.487090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.487078 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a08562e5\"" Oct 31 16:23:49.541642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.541554 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e722e8-0734-4f80-9cbb-7741658be1d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.541642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.541625 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmsfm\" (UniqueName: \"kubernetes.io/projected/83e722e8-0734-4f80-9cbb-7741658be1d5-kube-api-access-mmsfm\") pod \"payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.643919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.643877 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e722e8-0734-4f80-9cbb-7741658be1d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.644110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.643972 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmsfm\" (UniqueName: \"kubernetes.io/projected/83e722e8-0734-4f80-9cbb-7741658be1d5-kube-api-access-mmsfm\") pod \"payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.645360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.645331 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e722e8-0734-4f80-9cbb-7741658be1d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.672719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.672682 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:49.697336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.697302 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmsfm\" (UniqueName: \"kubernetes.io/projected/83e722e8-0734-4f80-9cbb-7741658be1d5-kube-api-access-mmsfm\") pod \"payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.712530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.710925 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:49.834314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.834153 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:49.985002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:49.984914 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:50.109289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:50.109212 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws"] Oct 31 16:23:50.113805 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:23:50.113777 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e722e8_0734_4f80_9cbb_7741658be1d5.slice/crio-b14fce66f4a2ef08d153c1a16732cb4035ad6039c8d8cfc88d5739eed5a05ba2 WatchSource:0}: Error finding container b14fce66f4a2ef08d153c1a16732cb4035ad6039c8d8cfc88d5739eed5a05ba2: Status 404 returned error can't find the container with id b14fce66f4a2ef08d153c1a16732cb4035ad6039c8d8cfc88d5739eed5a05ba2 Oct 31 16:23:50.173906 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:50.173875 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" event={"ID":"83e722e8-0734-4f80-9cbb-7741658be1d5","Type":"ContainerStarted","Data":"b14fce66f4a2ef08d153c1a16732cb4035ad6039c8d8cfc88d5739eed5a05ba2"} Oct 31 16:23:50.450100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:50.450067 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:50.595245 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:50.595214 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:51.514882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:51.514847 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7"] Oct 31 16:23:51.515356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:51.515085 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" containerID="cri-o://9b894f0dcca49e3c56cb89a0121ad888fdc2435a3000dffbd855a7f1011510d8" gracePeriod=30 Oct 31 16:23:51.736397 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:51.736356 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:51.904422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:51.904384 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:52.058945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:52.058905 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:52.886290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:52.886252 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:53.066776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:53.066731 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:53.199095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:53.199011 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" event={"ID":"83e722e8-0734-4f80-9cbb-7741658be1d5","Type":"ContainerStarted","Data":"16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57"} Oct 31 16:23:53.199095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:53.199077 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:23:53.226170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:53.226117 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:53.307210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:53.307150 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" podStartSLOduration=1.455947159 podStartE2EDuration="4.30711839s" totalImagesPullingTime="2.851171231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:23:53.304264802 +0000 UTC m=+1280.339559607" watchObservedRunningTime="2026-10-31 16:23:53.30711839 +0000 UTC m=+1280.342413195" Oct 31 16:23:53.384798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:53.384749 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:55.769713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:55.765580 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:55.940294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:55.939626 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:57.150218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.143734 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:57.278773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.278726 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:57.349026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.346219 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:57.482596 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.482506 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:57.582805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.582769 4880 generic.go:356] "Generic (PLEG): container finished" podID="ff17b788-4ed8-4981-bf66-7439999a184e" containerID="9b894f0dcca49e3c56cb89a0121ad888fdc2435a3000dffbd855a7f1011510d8" exitCode=0 Oct 31 16:23:57.601197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.583256 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" event={"ID":"ff17b788-4ed8-4981-bf66-7439999a184e","Type":"ContainerDied","Data":"9b894f0dcca49e3c56cb89a0121ad888fdc2435a3000dffbd855a7f1011510d8"} Oct 31 16:23:57.700653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.700615 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:23:57.960217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.840474 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:57.960217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:57.855427 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.004160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.000229 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 16:23:58.027252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.022393 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.167160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.163331 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 16:23:58.220216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.212580 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:23:58.221564 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.217208 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:58.297036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.295534 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.332162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.305303 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.368307 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.368260 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.471792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.410754 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:23:58.471792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.436570 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.488262 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.488215 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.550537 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.550480 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:58.682686 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.567347 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:23:58.718615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.706032 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:23:58.718615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.707242 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:23:58.914227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.880983 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 16:23:58.985301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:58.985277 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:23:59.028158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.011159 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 16:23:59.028158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.011473 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:23:59.029746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.028595 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.029746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.028884 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:23:59.059901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.052346 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:23:59.059901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.052400 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.224323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.224224 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 16:23:59.357241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.357182 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.375326 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.375286 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.727219 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.408548 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.411739 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.375568 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.726546 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.727913 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.519640 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.487823 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.489631 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.519056 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.664727 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.664770 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.735183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.707241 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.855246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.848447 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" event={"ID":"ff17b788-4ed8-4981-bf66-7439999a184e","Type":"ContainerDied","Data":"fb1f08f443bcdebe434c9bcf844cb414b227915cb80a96db5c1663fda80e26b4"} Oct 31 16:23:59.855569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.855547 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:23:59.855962 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.855946 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:23:59.865237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.855696 4880 scope.go:122] "RemoveContainer" containerID="9b894f0dcca49e3c56cb89a0121ad888fdc2435a3000dffbd855a7f1011510d8" Oct 31 16:23:59.875106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.875057 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-data\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.876168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.875318 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-socket\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.876491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.876472 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-kube-api-access-pbtzw\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-kube-api-access-pbtzw\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.876682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.876655 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-certs\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.880163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.876965 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-istio-token\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-istio-token\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.880438 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.880402 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-credential-socket\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.880598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.880582 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff17b788-4ed8-4981-bf66-7439999a184e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff17b788-4ed8-4981-bf66-7439999a184e-istio-podinfo\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.880806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.880790 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-envoy\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.880969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.880953 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff17b788-4ed8-4981-bf66-7439999a184e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff17b788-4ed8-4981-bf66-7439999a184e-istiod-ca-cert\") pod \"ff17b788-4ed8-4981-bf66-7439999a184e\" (UID: \"ff17b788-4ed8-4981-bf66-7439999a184e\") " Oct 31 16:23:59.890059 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.888235 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:59.905612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.905580 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:23:59.945665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.945518 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:23:59.990574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.990477 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff17b788-4ed8-4981-bf66-7439999a184e-istiod-ca-cert" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.995325 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-certs" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:23:59.995572 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-socket" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.005339 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-credential-socket" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.052254 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-data" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.052407 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.054302 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.054328 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff17b788-4ed8-4981-bf66-7439999a184e-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.054344 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.054359 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.070606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.054374 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.093756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.074418 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-istio-token" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:24:00.093756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.075212 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 16:24:00.133644 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.133611 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:00.164505 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.164474 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.239189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.235033 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 16:24:00.239836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.239804 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff17b788-4ed8-4981-bf66-7439999a184e-istio-podinfo" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:24:00.251412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.246210 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" podUID="14405e4c-cada-487b-89a2-f770fcb8c907" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:00.272165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.261294 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-envoy" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:24:00.318167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.287570 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff17b788-4ed8-4981-bf66-7439999a184e-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.318469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.318439 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff17b788-4ed8-4981-bf66-7439999a184e-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.358170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.358033 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:00.365467 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.362213 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:24:00.365467 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.362267 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:00.457427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.457369 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-kube-api-access-pbtzw" pod "ff17b788-4ed8-4981-bf66-7439999a184e" (UID: "ff17b788-4ed8-4981-bf66-7439999a184e"). InnerVolumeSpecName "kube-api-access-pbtzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:24:00.483493 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.483465 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtzw\" (UniqueName: \"kubernetes.io/projected/ff17b788-4ed8-4981-bf66-7439999a184e-kube-api-access-pbtzw\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:24:00.696332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.669847 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7" Oct 31 16:24:00.738413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.738373 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:00.871387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.761197 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:00.871387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.761317 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 16:24:00.871387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.869251 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:24:00.871387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.761343 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:24:00.877505 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.761377 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:24:00.891308 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.761416 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:24:00.891817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:00.891787 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:01.054786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:01.052830 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 16:24:01.082166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:01.080262 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:01.257169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:01.235304 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:01.585211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:01.584290 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:01.654162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:01.587396 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:24:01.654162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:01.587451 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:01.761161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:01.687967 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 16:24:02.021284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.021232 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:24:02.070598 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:24:02.070546 4880 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff17b788_4ed8_4981_bf66_7439999a184e.slice/crio-fb1f08f443bcdebe434c9bcf844cb414b227915cb80a96db5c1663fda80e26b4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff17b788_4ed8_4981_bf66_7439999a184e.slice\": RecentStats: unable to find data in memory cache]" Oct 31 16:24:02.152377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.152330 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:02.334222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.316150 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:02.608215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.551429 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 31 16:24:02.654927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.157151 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.157268 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.316930 4880 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-bthm6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.697221 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-bthm6" podUID="e9cad0b0-2b17-48f6-9c14-12a419791b1f" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.317272 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.697628 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.394475 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.157010 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Oct 31 16:24:02.698165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.156874 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:02.706479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.394368 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 31 16:24:02.706479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.697165 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 16:24:02.721257 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.679000 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:02.938744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.906913 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:24:02.938744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:02.907292 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 31 16:24:03.077126 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.077047 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:03.180290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.180243 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:03.391671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.389579 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7"] Oct 31 16:24:03.391671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.389614 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-zvlv7"] Oct 31 16:24:03.396821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.396800 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:24:03.397390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.397366 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:24:03.398214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.398043 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:24:03.400337 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.400317 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:24:03.401216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.400900 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:24:03.401424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.401402 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:24:03.401504 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.401472 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:24:03.406288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.406269 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:24:03.406449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.406436 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:24:03.429017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:03.428978 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:24:03.891609 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:24:03.891565 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:24:04.208848 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:04.208777 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:24:05.091707 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:05.091679 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" path="/var/lib/kubelet/pods/ff17b788-4ed8-4981-bf66-7439999a184e/volumes" Oct 31 16:24:05.160863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:05.160824 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:06.120586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:06.120553 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:06.291768 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:06.291726 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:06.450434 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:06.450397 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:10.045127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:10.045093 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:11.893848 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:11.893811 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:14.869354 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:14.869324 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:16.310533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:16.310499 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:19.659687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:19.659646 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:22.236801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:22.236767 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:24.426787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:24.426755 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:28.887448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.887412 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz"] Oct 31 16:24:28.888352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.888219 4880 cpu_manager.go:414] "containerMap: removing container" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" Oct 31 16:24:28.888352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.888249 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" Oct 31 16:24:28.888563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.888526 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff17b788-4ed8-4981-bf66-7439999a184e" containerName="istio-proxy" Oct 31 16:24:28.895811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.895790 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:28.943679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.943657 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz"] Oct 31 16:24:28.985252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.985231 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f27146d0-serving-cert\"" Oct 31 16:24:28.999918 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.999891 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-metrics-certs\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.000026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:28.999939 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtfnv\" (UniqueName: \"kubernetes.io/projected/3cd10a17-7353-482d-b9d6-be689c4100d9-kube-api-access-gtfnv\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.000172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.000123 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.000223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.000195 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd10a17-7353-482d-b9d6-be689c4100d9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.089586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.089552 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:29.102005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.101981 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.102175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.102018 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd10a17-7353-482d-b9d6-be689c4100d9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.102175 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:24:29.102112 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-serving-cert: secret "maas-api-e2e-migrate-f27146d0-serving-cert" not found Oct 31 16:24:29.102175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.102114 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-metrics-certs\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.102359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.102193 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtfnv\" (UniqueName: \"kubernetes.io/projected/3cd10a17-7353-482d-b9d6-be689c4100d9-kube-api-access-gtfnv\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.102359 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:24:29.102323 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls podName:3cd10a17-7353-482d-b9d6-be689c4100d9 nodeName:}" failed. No retries permitted until 2026-10-31 16:24:29.602301773 +0000 UTC m=+1316.637596564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls") pod "maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" (UID: "3cd10a17-7353-482d-b9d6-be689c4100d9") : secret "maas-api-e2e-migrate-f27146d0-serving-cert" not found Oct 31 16:24:29.102762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.102740 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd10a17-7353-482d-b9d6-be689c4100d9-openshift-service-ca\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.104462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.104445 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-metrics-certs\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.158055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.157914 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtfnv\" (UniqueName: \"kubernetes.io/projected/3cd10a17-7353-482d-b9d6-be689c4100d9-kube-api-access-gtfnv\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.265719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.265681 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:29.417849 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.417720 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:29.609332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.609300 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.611712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.611688 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls\") pod \"maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:29.810090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:29.810017 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:30.186714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:30.186681 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz"] Oct 31 16:24:30.191258 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:24:30.191231 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd10a17_7353_482d_b9d6_be689c4100d9.slice/crio-151acb5193f6b14571aea66b7bc086e50eeadc28e22f9ff92267203527da59a0 WatchSource:0}: Error finding container 151acb5193f6b14571aea66b7bc086e50eeadc28e22f9ff92267203527da59a0: Status 404 returned error can't find the container with id 151acb5193f6b14571aea66b7bc086e50eeadc28e22f9ff92267203527da59a0 Oct 31 16:24:30.309451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:30.309416 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" event={"ID":"3cd10a17-7353-482d-b9d6-be689c4100d9","Type":"ContainerStarted","Data":"151acb5193f6b14571aea66b7bc086e50eeadc28e22f9ff92267203527da59a0"} Oct 31 16:24:30.404946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:30.404909 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:32.332771 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:32.332732 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" event={"ID":"3cd10a17-7353-482d-b9d6-be689c4100d9","Type":"ContainerStarted","Data":"2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5"} Oct 31 16:24:32.333180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:32.332801 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:32.441983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:32.441932 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" podStartSLOduration=3.38377191 podStartE2EDuration="4.441918774s" totalImagesPullingTime="1.058146864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:32.438743996 +0000 UTC m=+1319.474038800" watchObservedRunningTime="2026-10-31 16:24:32.441918774 +0000 UTC m=+1319.477213577" Oct 31 16:24:33.521783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.521748 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg"] Oct 31 16:24:33.526558 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.526535 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.621000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.620923 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a08562e5-serving-cert\"" Oct 31 16:24:33.625228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.625207 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg"] Oct 31 16:24:33.680477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.680445 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jxt\" (UniqueName: \"kubernetes.io/projected/b7baf31b-91da-426e-adc2-ed181c9bfcc8-kube-api-access-c2jxt\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.680661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.680505 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7baf31b-91da-426e-adc2-ed181c9bfcc8-openshift-service-ca\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.680661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.680631 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.680889 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.680857 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-metrics-certs\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.753725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.753692 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:33.782467 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.782431 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-metrics-certs\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.782645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.782555 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jxt\" (UniqueName: \"kubernetes.io/projected/b7baf31b-91da-426e-adc2-ed181c9bfcc8-kube-api-access-c2jxt\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.782645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.782623 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7baf31b-91da-426e-adc2-ed181c9bfcc8-openshift-service-ca\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.782733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.782718 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.782872 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:24:33.782855 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-serving-cert: secret "maas-api-e2e-ait-a08562e5-serving-cert" not found Oct 31 16:24:33.782940 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:24:33.782930 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls podName:b7baf31b-91da-426e-adc2-ed181c9bfcc8 nodeName:}" failed. No retries permitted until 2026-10-31 16:24:34.282911507 +0000 UTC m=+1321.318206301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls") pod "maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" (UID: "b7baf31b-91da-426e-adc2-ed181c9bfcc8") : secret "maas-api-e2e-ait-a08562e5-serving-cert" not found Oct 31 16:24:33.783406 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.783382 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7baf31b-91da-426e-adc2-ed181c9bfcc8-openshift-service-ca\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.785074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.785047 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-metrics-certs\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.837631 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.837603 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jxt\" (UniqueName: \"kubernetes.io/projected/b7baf31b-91da-426e-adc2-ed181c9bfcc8-kube-api-access-c2jxt\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:33.888454 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:24:33.885877 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:24:33.913686 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:33.913647 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:34.063707 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:34.063670 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:34.289848 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:34.289760 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:34.292710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:34.292685 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls\") pod \"maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:34.439912 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:34.439878 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:34.813480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:34.813445 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg"] Oct 31 16:24:34.818405 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:24:34.818377 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7baf31b_91da_426e_adc2_ed181c9bfcc8.slice/crio-b76e5ed333c354c1f443599cd5f55198c65db8eb399fff6d3a46f20e84ea5c45 WatchSource:0}: Error finding container b76e5ed333c354c1f443599cd5f55198c65db8eb399fff6d3a46f20e84ea5c45: Status 404 returned error can't find the container with id b76e5ed333c354c1f443599cd5f55198c65db8eb399fff6d3a46f20e84ea5c45 Oct 31 16:24:35.355508 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:35.355474 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" event={"ID":"b7baf31b-91da-426e-adc2-ed181c9bfcc8","Type":"ContainerStarted","Data":"b76e5ed333c354c1f443599cd5f55198c65db8eb399fff6d3a46f20e84ea5c45"} Oct 31 16:24:36.368228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:36.368174 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" event={"ID":"b7baf31b-91da-426e-adc2-ed181c9bfcc8","Type":"ContainerStarted","Data":"5d331f95be2b66fbb93ad76e2e722bb45bd8f59dd246ba3c99f44a63ac3991bc"} Oct 31 16:24:36.368740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:36.368296 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:36.480437 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:36.480373 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" podStartSLOduration=2.653496123 podStartE2EDuration="3.480351069s" totalImagesPullingTime="826.854946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:36.474753411 +0000 UTC m=+1323.510048215" watchObservedRunningTime="2026-10-31 16:24:36.480351069 +0000 UTC m=+1323.515645876" Oct 31 16:24:38.435161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.433350 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:24:38.445814 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.440291 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:24:38.477331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.477244 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:38.589410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.588227 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:38.842047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.840816 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:38.842047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.841275 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:24:38.864772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.864728 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:38.885589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.885527 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:38.886276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:38.886241 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.079159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.055241 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.191340 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.190222 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.227260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.227217 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.419288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.419242 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Oct 31 16:24:39.429692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.429663 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp"] Oct 31 16:24:39.436996 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.436236 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.455576 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.455501 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 31 16:24:39.600275 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.600234 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.602173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.602110 4880 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Oct 31 16:24:39.846273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.846176 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:39.896318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.896264 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.937159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.912165 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:39.937159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.914804 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 16:24:39.937159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:39.916573 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:40.099215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.056121 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:40.194242 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.179690 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:40.207932 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.207894 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-xrfgk" podUID="14405e4c-cada-487b-89a2-f770fcb8c907" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.43:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:40.293161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.229226 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Oct 31 16:24:40.336155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.317383 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:40.354280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.354194 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:24:40.385198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.368512 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:24:40.421125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.421089 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:24:40.421641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.421623 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:24:40.427828 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.427808 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:24:40.468779 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.468750 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp"] Oct 31 16:24:40.472446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.472414 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:24:40.472685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.472668 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:24:40.472879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.472864 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:24:40.473045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.473031 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:24:40.473241 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.473226 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:24:40.473799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.372886 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.474752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.472278 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:40.476950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.476932 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:24:40.478019 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.477991 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:40.480215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.480188 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 16:24:40.491908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.482415 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:24:40.491908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.485867 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:24:40.532529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.526458 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-caa664e1-serving-cert\"" Oct 31 16:24:40.570230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.569698 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:24:40.572478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.571998 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:24:40.575539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.575513 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:24:40.603181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.602476 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:40.607975 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.607905 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e96e552-44b8-4238-ae01-1a6002cf83aa-openshift-service-ca\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.608083 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.608049 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfql4\" (UniqueName: \"kubernetes.io/projected/2e96e552-44b8-4238-ae01-1a6002cf83aa-kube-api-access-jfql4\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.608166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.608080 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-metrics-certs\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.613344 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.613321 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-maas-api-tls\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.613477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.613460 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:24:40.642789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.642763 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:24:40.667365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.667109 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:24:40.675756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.675709 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:24:40.715699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.715664 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfql4\" (UniqueName: \"kubernetes.io/projected/2e96e552-44b8-4238-ae01-1a6002cf83aa-kube-api-access-jfql4\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.715960 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.715937 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-metrics-certs\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.716332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.716310 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-maas-api-tls\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.716515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.716491 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e96e552-44b8-4238-ae01-1a6002cf83aa-openshift-service-ca\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.717276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.717256 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e96e552-44b8-4238-ae01-1a6002cf83aa-openshift-service-ca\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.720627 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.720603 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-maas-api-tls\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.721938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.721909 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-metrics-certs\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.775183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.774419 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfql4\" (UniqueName: \"kubernetes.io/projected/2e96e552-44b8-4238-ae01-1a6002cf83aa-kube-api-access-jfql4\") pod \"maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:40.848987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:40.848956 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:41.255595 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:41.255562 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp"] Oct 31 16:24:41.257235 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:24:41.257203 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e96e552_44b8_4238_ae01_1a6002cf83aa.slice/crio-ee0e25141bc2504a51e40fe9990eec0b1ac9a55bf55dc296d210e20caff4d1fe WatchSource:0}: Error finding container ee0e25141bc2504a51e40fe9990eec0b1ac9a55bf55dc296d210e20caff4d1fe: Status 404 returned error can't find the container with id ee0e25141bc2504a51e40fe9990eec0b1ac9a55bf55dc296d210e20caff4d1fe Oct 31 16:24:42.234005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:42.233966 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" event={"ID":"2e96e552-44b8-4238-ae01-1a6002cf83aa","Type":"ContainerStarted","Data":"ee0e25141bc2504a51e40fe9990eec0b1ac9a55bf55dc296d210e20caff4d1fe"} Oct 31 16:24:42.392563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:42.392535 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:24:43.245045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:43.245008 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" event={"ID":"2e96e552-44b8-4238-ae01-1a6002cf83aa","Type":"ContainerStarted","Data":"4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710"} Oct 31 16:24:43.245475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:43.245125 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:43.357680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:43.357616 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" podStartSLOduration=3.496343884 podStartE2EDuration="4.357596574s" totalImagesPullingTime="861.25269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:24:43.353376586 +0000 UTC m=+1330.388671389" watchObservedRunningTime="2026-10-31 16:24:43.357596574 +0000 UTC m=+1330.392891379" Oct 31 16:24:44.785982 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:44.785926 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:49.258085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:49.258057 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:24:49.552915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:49.552845 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:54.465633 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:54.465595 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:24:59.286011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:24:59.285972 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:03.881842 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:25:03.881809 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:25:04.069579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:04.069552 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:07.191745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.191705 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf"] Oct 31 16:25:07.196566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.196542 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258108 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258168 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258271 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258364 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258335 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn52n\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-kube-api-access-fn52n\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258366 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258389 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258507 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258574 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.258652 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.258636 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360298 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360377 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn52n\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-kube-api-access-fn52n\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360413 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360445 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360512 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360603 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360663 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360710 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360663 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360807 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360838 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.360904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.360844 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.361172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.361152 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.361242 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.361231 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.362788 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.362765 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.362973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.362953 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.387216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.387192 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf"] Oct 31 16:25:07.413099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.413070 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.413312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.413294 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn52n\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-kube-api-access-fn52n\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggxdf\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.510497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.510432 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:07.940244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.940186 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf"] Oct 31 16:25:07.944943 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:25:07.944914 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19ed2ae_c6c2_455f_b835_2b59a9d5f461.slice/crio-017cada8c6ade42dd3e6f44e952b1547f8eb6ef850499bf60463b8312959d2a3 WatchSource:0}: Error finding container 017cada8c6ade42dd3e6f44e952b1547f8eb6ef850499bf60463b8312959d2a3: Status 404 returned error can't find the container with id 017cada8c6ade42dd3e6f44e952b1547f8eb6ef850499bf60463b8312959d2a3 Oct 31 16:25:07.947545 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.947513 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:07.947629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.947578 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:07.947629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:07.947625 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:08.438592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:08.438558 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" event={"ID":"e19ed2ae-c6c2-455f-b835-2b59a9d5f461","Type":"ContainerStarted","Data":"9ff870fa4c5bd4213af3993645752b4f99142070d452bd60a2044a3126d93f09"} Oct 31 16:25:08.439070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:08.438610 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" event={"ID":"e19ed2ae-c6c2-455f-b835-2b59a9d5f461","Type":"ContainerStarted","Data":"017cada8c6ade42dd3e6f44e952b1547f8eb6ef850499bf60463b8312959d2a3"} Oct 31 16:25:08.514243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:08.514205 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:08.651661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:08.651577 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podStartSLOduration=1.651563322 podStartE2EDuration="1.651563322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:08.593383721 +0000 UTC m=+1355.628678527" watchObservedRunningTime="2026-10-31 16:25:08.651563322 +0000 UTC m=+1355.686858125" Oct 31 16:25:08.949615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:08.949580 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:09.512096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:09.512061 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 16:25:09.512508 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:09.512128 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 16:25:10.511492 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:10.511462 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 16:25:10.511681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:10.511516 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 16:25:11.511301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:11.511269 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 16:25:11.511694 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:11.511362 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 16:25:12.510939 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:12.510905 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 16:25:12.511130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:12.510967 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 16:25:13.511106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:13.511075 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 16:25:13.511511 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:13.511149 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 16:25:13.808719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:13.808641 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:14.511381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:14.511349 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 16:25:14.511784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:14.511406 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 16:25:16.201877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:16.201848 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:16.202418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:16.202396 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:16.203234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:16.203207 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:25:18.645602 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:18.645557 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:22.276230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.276185 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph"] Oct 31 16:25:22.288983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.288950 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324348 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324408 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324450 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324497 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324537 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324603 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324642 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324710 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3059189-8881-400a-9fa4-4cef8b277a8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.324739 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wr9x\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-kube-api-access-4wr9x\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.425336 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph"] Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.426805 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.426866 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.426940 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.426980 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.427055 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3059189-8881-400a-9fa4-4cef8b277a8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.427085 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wr9x\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-kube-api-access-4wr9x\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.427291 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.427346 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.427386 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.427826 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.428358 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.428987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.428652 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.432323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.432295 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3059189-8881-400a-9fa4-4cef8b277a8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.433449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.432516 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.437811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.437788 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.439405 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.439383 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.509451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.509358 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.511226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.511175 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wr9x\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-kube-api-access-4wr9x\") pod \"maas-default-gateway-openshift-default-fcb75c45f-t7qph\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:22.666568 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:22.660454 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:23.122524 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:25:23.122495 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3059189_8881_400a_9fa4_4cef8b277a8d.slice/crio-2a232e9f29a545a3a4573ef21a9af15fcf1696442fca13a71c2b3e7015e2907e WatchSource:0}: Error finding container 2a232e9f29a545a3a4573ef21a9af15fcf1696442fca13a71c2b3e7015e2907e: Status 404 returned error can't find the container with id 2a232e9f29a545a3a4573ef21a9af15fcf1696442fca13a71c2b3e7015e2907e Oct 31 16:25:23.124998 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.124962 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:23.125106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.125038 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:23.125106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.125068 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:23.139919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.139893 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph"] Oct 31 16:25:23.546988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.546946 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:23.590642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.590605 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" event={"ID":"f3059189-8881-400a-9fa4-4cef8b277a8d","Type":"ContainerStarted","Data":"8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308"} Oct 31 16:25:23.590817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.590671 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" event={"ID":"f3059189-8881-400a-9fa4-4cef8b277a8d","Type":"ContainerStarted","Data":"2a232e9f29a545a3a4573ef21a9af15fcf1696442fca13a71c2b3e7015e2907e"} Oct 31 16:25:23.667169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.667128 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:23.705538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:23.705490 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podStartSLOduration=1.705476193 podStartE2EDuration="1.705476193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:23.701322927 +0000 UTC m=+1370.736617731" watchObservedRunningTime="2026-10-31 16:25:23.705476193 +0000 UTC m=+1370.740770996" Oct 31 16:25:24.668097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:24.668063 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 16:25:24.668567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:24.668125 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 16:25:25.667040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:25.667000 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 16:25:25.667235 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:25.667057 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 16:25:26.666700 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:26.666659 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 16:25:26.667168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:26.666724 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 16:25:27.667356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:27.667311 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 16:25:27.667753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:27.667378 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 16:25:28.357518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:28.357481 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:28.667127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:28.667095 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 16:25:28.667407 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:28.667382 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 16:25:29.667695 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:29.667656 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 16:25:29.668209 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:29.667713 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 16:25:30.546304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.546270 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr"] Oct 31 16:25:30.560997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.560963 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.667445 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.667404 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 16:25:30.667638 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.667474 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 16:25:30.702073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702033 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8bc1761-7028-4947-b122-ed7dbc238736-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702107 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702238 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702272 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2pqk\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-kube-api-access-t2pqk\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702359 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8bc1761-7028-4947-b122-ed7dbc238736-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702439 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702486 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702591 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.702715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.702653 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.718510 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.718478 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-caa664e1-openshift-default-dockercfg-cbxhc\"" Oct 31 16:25:30.751444 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.751404 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr"] Oct 31 16:25:30.757306 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.757244 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:30.804775 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.804684 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.804940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.804773 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8bc1761-7028-4947-b122-ed7dbc238736-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.804940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.804829 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.805073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.804943 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.805073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.804975 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2pqk\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-kube-api-access-t2pqk\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.805073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.805066 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8bc1761-7028-4947-b122-ed7dbc238736-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.805251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.805165 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.805251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.805211 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.805352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.805317 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.805861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.805836 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.806660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.806632 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.807026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.806999 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.807111 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.807050 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8bc1761-7028-4947-b122-ed7dbc238736-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.807425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.807401 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.809842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.809814 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.813907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.813863 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8bc1761-7028-4947-b122-ed7dbc238736-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.858490 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.858461 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.858744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.858718 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2pqk\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-kube-api-access-t2pqk\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.887551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.887519 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:30.942756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:30.942725 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:31.338673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.338517 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr"] Oct 31 16:25:31.341753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.341708 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:31.341869 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.341803 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:31.341869 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.341850 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:25:31.702160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.701473 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" event={"ID":"d8bc1761-7028-4947-b122-ed7dbc238736","Type":"ContainerStarted","Data":"a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41"} Oct 31 16:25:31.702160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.701544 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" event={"ID":"d8bc1761-7028-4947-b122-ed7dbc238736","Type":"ContainerStarted","Data":"a84c849ce15e211a53709c789619daf1993e3841c0dbd5a8be8e480eda90bd10"} Oct 31 16:25:31.813159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.812876 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" podStartSLOduration=1.812855566 podStartE2EDuration="1.812855566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:25:31.808846064 +0000 UTC m=+1378.844140864" watchObservedRunningTime="2026-10-31 16:25:31.812855566 +0000 UTC m=+1378.848150372" Oct 31 16:25:31.888160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:31.887691 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:32.289785 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:32.289757 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:32.290781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:32.290760 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:32.291911 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:32.291889 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:25:32.889506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:32.889473 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 16:25:32.889905 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:32.889530 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 16:25:33.884700 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:25:33.884666 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:25:33.888153 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:33.888117 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 16:25:33.888284 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:33.888180 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 16:25:34.016305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:34.016269 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:34.888613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:34.888579 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 16:25:34.888786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:34.888650 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 16:25:35.888425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:35.888388 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 16:25:35.888914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:35.888447 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 16:25:36.888798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:36.888767 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 31 16:25:36.889265 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:36.888824 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 31 16:25:37.893310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:37.893281 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:37.893794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:37.893727 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:37.894410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:37.894389 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:25:39.470969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:39.470935 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:43.639421 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:43.639345 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:43.826031 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:43.825989 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:44.661242 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:44.661209 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:46.447902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:46.447856 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:46.597714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:46.597678 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:47.786070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:47.786035 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:48.952110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:48.952077 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws"] Oct 31 16:25:49.217827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:49.217749 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:49.450231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:49.450197 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:50.005368 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:50.005324 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:51.541477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:51.541446 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:51.692862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:51.692826 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:53.952797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:53.952756 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" podUID="83e722e8-0734-4f80-9cbb-7741658be1d5" containerName="payload-pre-processing" containerID="cri-o://16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57" gracePeriod=25 Oct 31 16:25:54.230667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.230639 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:25:54.256361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.256334 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e722e8-0734-4f80-9cbb-7741658be1d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e722e8-0734-4f80-9cbb-7741658be1d5-plugins-config-volume\") pod \"83e722e8-0734-4f80-9cbb-7741658be1d5\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " Oct 31 16:25:54.256479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.256403 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e722e8-0734-4f80-9cbb-7741658be1d5-kube-api-access-mmsfm\" (UniqueName: \"kubernetes.io/projected/83e722e8-0734-4f80-9cbb-7741658be1d5-kube-api-access-mmsfm\") pod \"83e722e8-0734-4f80-9cbb-7741658be1d5\" (UID: \"83e722e8-0734-4f80-9cbb-7741658be1d5\") " Oct 31 16:25:54.256786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.256759 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e722e8-0734-4f80-9cbb-7741658be1d5-plugins-config-volume" pod "83e722e8-0734-4f80-9cbb-7741658be1d5" (UID: "83e722e8-0734-4f80-9cbb-7741658be1d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:25:54.257274 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.257256 4880 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83e722e8-0734-4f80-9cbb-7741658be1d5-plugins-config-volume\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:25:54.258565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.258547 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e722e8-0734-4f80-9cbb-7741658be1d5-kube-api-access-mmsfm" pod "83e722e8-0734-4f80-9cbb-7741658be1d5" (UID: "83e722e8-0734-4f80-9cbb-7741658be1d5"). InnerVolumeSpecName "kube-api-access-mmsfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:25:54.358856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.358824 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmsfm\" (UniqueName: \"kubernetes.io/projected/83e722e8-0734-4f80-9cbb-7741658be1d5-kube-api-access-mmsfm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:25:54.937515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.937479 4880 generic.go:356] "Generic (PLEG): container finished" podID="83e722e8-0734-4f80-9cbb-7741658be1d5" containerID="16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57" exitCode=0 Oct 31 16:25:54.937674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.937523 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" Oct 31 16:25:54.937674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.937559 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" event={"ID":"83e722e8-0734-4f80-9cbb-7741658be1d5","Type":"ContainerDied","Data":"16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57"} Oct 31 16:25:54.937674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.937616 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" event={"ID":"83e722e8-0734-4f80-9cbb-7741658be1d5","Type":"ContainerDied","Data":"b14fce66f4a2ef08d153c1a16732cb4035ad6039c8d8cfc88d5739eed5a05ba2"} Oct 31 16:25:54.937674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.937639 4880 scope.go:122] "RemoveContainer" containerID="16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57" Oct 31 16:25:54.949593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.949560 4880 scope.go:122] "RemoveContainer" containerID="16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57" Oct 31 16:25:54.949830 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:25:54.949807 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57\": container with ID starting with 16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57 not found: ID does not exist" containerID="16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57" Oct 31 16:25:54.949878 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:54.949840 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57"} err="failed to get container status \"16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57\": rpc error: code = NotFound desc = could not find container \"16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57\": container with ID starting with 16a9d6fee0b34e7a143043d9238ea86c96181042b1ed26eebd976afcda3f2a57 not found: ID does not exist" Oct 31 16:25:55.092484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:55.092456 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws"] Oct 31 16:25:55.096313 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:55.096290 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws"] Oct 31 16:25:55.207489 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:55.207401 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-a08562e5-c6f5dd659-nd2ws" podUID="83e722e8-0734-4f80-9cbb-7741658be1d5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.99:50052\" within 1s: context deadline exceeded" Oct 31 16:25:55.207640 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:25:55.207488 4880 logging.go:55] [core] [Channel #56 SubChannel #57]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.99:50052", ServerName: "10.132.0.99:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.99:50052: operation was canceled" Oct 31 16:25:55.678750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:55.678708 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:55.825913 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:55.825880 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:25:57.091410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:57.091382 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e722e8-0734-4f80-9cbb-7741658be1d5" path="/var/lib/kubelet/pods/83e722e8-0734-4f80-9cbb-7741658be1d5/volumes" Oct 31 16:25:59.854903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:25:59.854872 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:03.168835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:03.168798 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:03.881852 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:26:03.881813 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:26:04.673646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:04.673612 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:09.007937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:09.007895 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:09.561927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:09.561891 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:14.567037 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:14.566997 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:15.902175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:15.902123 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l"] Oct 31 16:26:15.903254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:15.903229 4880 cpu_manager.go:414] "containerMap: removing container" podUID="83e722e8-0734-4f80-9cbb-7741658be1d5" containerName="payload-pre-processing" Oct 31 16:26:15.903254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:15.903255 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e722e8-0734-4f80-9cbb-7741658be1d5" containerName="payload-pre-processing" Oct 31 16:26:15.903437 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:15.903412 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e722e8-0734-4f80-9cbb-7741658be1d5" containerName="payload-pre-processing" Oct 31 16:26:15.908651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:15.908631 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.035726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.035688 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.035726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.035735 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7c9q\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-kube-api-access-k7c9q\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.035984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.035894 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.035984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.035945 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.036099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.036074 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.036183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.036103 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.036280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.036262 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.036342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.036314 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.036486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.036464 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.103927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.103896 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l"] Oct 31 16:26:16.138456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138421 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138477 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138530 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138549 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138660 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138716 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138837 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138892 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-certs\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.138971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138922 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-data\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.139085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.138979 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.139085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.139011 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7c9q\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-kube-api-access-k7c9q\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.139085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.139026 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-credential-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.139682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.139657 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-socket\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.139822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.139804 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istiod-ca-cert\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.141678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.141646 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-envoy\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.141797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.141696 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-podinfo\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.191724 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.191662 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-token\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.192009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.191987 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7c9q\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-kube-api-access-k7c9q\") pod \"e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.222770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.222739 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:16.667048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.667022 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l"] Oct 31 16:26:16.670549 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:26:16.670519 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63a05d2_61b8_4a02_89bf_9146716c9d5b.slice/crio-7ed525bc1cbe22f797b19de88921d5f9756f556350554f8c4814092117026921 WatchSource:0}: Error finding container 7ed525bc1cbe22f797b19de88921d5f9756f556350554f8c4814092117026921: Status 404 returned error can't find the container with id 7ed525bc1cbe22f797b19de88921d5f9756f556350554f8c4814092117026921 Oct 31 16:26:16.673862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.673821 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:16.673948 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.673911 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:16.673985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:16.673956 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:17.134587 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:17.134551 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" event={"ID":"f63a05d2-61b8-4a02-89bf-9146716c9d5b","Type":"ContainerStarted","Data":"b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76"} Oct 31 16:26:17.135051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:17.134615 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" event={"ID":"f63a05d2-61b8-4a02-89bf-9146716c9d5b","Type":"ContainerStarted","Data":"7ed525bc1cbe22f797b19de88921d5f9756f556350554f8c4814092117026921"} Oct 31 16:26:17.231160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:17.226975 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:17.231358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:17.231244 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 16:26:17.231358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:17.231298 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 16:26:17.348329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:17.348114 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podStartSLOduration=2.348093776 podStartE2EDuration="2.348093776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:17.295566472 +0000 UTC m=+1424.330861277" watchObservedRunningTime="2026-10-31 16:26:17.348093776 +0000 UTC m=+1424.383388582" Oct 31 16:26:18.243168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:18.242219 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 16:26:18.243168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:18.242280 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 16:26:18.719322 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:18.719292 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz"] Oct 31 16:26:18.719495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:18.719470 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" podUID="3cd10a17-7353-482d-b9d6-be689c4100d9" containerName="maas-api" containerID="cri-o://2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5" gracePeriod=30 Oct 31 16:26:18.846171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:18.845446 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:19.202670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.202293 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:26:19.204231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.203737 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd10a17-7353-482d-b9d6-be689c4100d9-kube-api-access-gtfnv\" (UniqueName: \"kubernetes.io/projected/3cd10a17-7353-482d-b9d6-be689c4100d9-kube-api-access-gtfnv\") pod \"3cd10a17-7353-482d-b9d6-be689c4100d9\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " Oct 31 16:26:19.204231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.203877 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-metrics-certs\") pod \"3cd10a17-7353-482d-b9d6-be689c4100d9\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " Oct 31 16:26:19.204231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.203995 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls\") pod \"3cd10a17-7353-482d-b9d6-be689c4100d9\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " Oct 31 16:26:19.204231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.204046 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd10a17-7353-482d-b9d6-be689c4100d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd10a17-7353-482d-b9d6-be689c4100d9-openshift-service-ca\") pod \"3cd10a17-7353-482d-b9d6-be689c4100d9\" (UID: \"3cd10a17-7353-482d-b9d6-be689c4100d9\") " Oct 31 16:26:19.204773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.204574 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd10a17-7353-482d-b9d6-be689c4100d9-openshift-service-ca" pod "3cd10a17-7353-482d-b9d6-be689c4100d9" (UID: "3cd10a17-7353-482d-b9d6-be689c4100d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:19.206228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.206210 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd10a17-7353-482d-b9d6-be689c4100d9-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:19.206531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.206510 4880 generic.go:356] "Generic (PLEG): container finished" podID="3cd10a17-7353-482d-b9d6-be689c4100d9" containerID="2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5" exitCode=0 Oct 31 16:26:19.207196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.206642 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" event={"ID":"3cd10a17-7353-482d-b9d6-be689c4100d9","Type":"ContainerDied","Data":"2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5"} Oct 31 16:26:19.207196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.206995 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" event={"ID":"3cd10a17-7353-482d-b9d6-be689c4100d9","Type":"ContainerDied","Data":"151acb5193f6b14571aea66b7bc086e50eeadc28e22f9ff92267203527da59a0"} Oct 31 16:26:19.207196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.207066 4880 scope.go:122] "RemoveContainer" containerID="2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5" Oct 31 16:26:19.207488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.207454 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls" pod "3cd10a17-7353-482d-b9d6-be689c4100d9" (UID: "3cd10a17-7353-482d-b9d6-be689c4100d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:19.222173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.219635 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd10a17-7353-482d-b9d6-be689c4100d9-kube-api-access-gtfnv" pod "3cd10a17-7353-482d-b9d6-be689c4100d9" (UID: "3cd10a17-7353-482d-b9d6-be689c4100d9"). InnerVolumeSpecName "kube-api-access-gtfnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:19.227747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.226489 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 16:26:19.227747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.226538 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 16:26:19.242157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.237342 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-metrics-certs" pod "3cd10a17-7353-482d-b9d6-be689c4100d9" (UID: "3cd10a17-7353-482d-b9d6-be689c4100d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:19.298446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.297869 4880 scope.go:122] "RemoveContainer" containerID="2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5" Oct 31 16:26:19.298446 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:26:19.298201 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5\": container with ID starting with 2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5 not found: ID does not exist" containerID="2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5" Oct 31 16:26:19.298446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.298231 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5"} err="failed to get container status \"2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5\": rpc error: code = NotFound desc = could not find container \"2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5\": container with ID starting with 2c27db8e10265c5415be4869d5c9cd997bdd9555a630a6799acf62c7a179e0a5 not found: ID does not exist" Oct 31 16:26:19.308248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.307587 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:19.308248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.307609 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd10a17-7353-482d-b9d6-be689c4100d9-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:19.308248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.307619 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtfnv\" (UniqueName: \"kubernetes.io/projected/3cd10a17-7353-482d-b9d6-be689c4100d9-kube-api-access-gtfnv\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:19.899875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.899841 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg"] Oct 31 16:26:19.900074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:19.900049 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" podUID="b7baf31b-91da-426e-adc2-ed181c9bfcc8" containerName="maas-api" containerID="cri-o://5d331f95be2b66fbb93ad76e2e722bb45bd8f59dd246ba3c99f44a63ac3991bc" gracePeriod=30 Oct 31 16:26:20.052599 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.052500 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:20.232159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.231654 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 16:26:20.232159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.231723 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 16:26:20.269159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.268780 4880 generic.go:356] "Generic (PLEG): container finished" podID="b7baf31b-91da-426e-adc2-ed181c9bfcc8" containerID="5d331f95be2b66fbb93ad76e2e722bb45bd8f59dd246ba3c99f44a63ac3991bc" exitCode=0 Oct 31 16:26:20.269159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.268881 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" event={"ID":"b7baf31b-91da-426e-adc2-ed181c9bfcc8","Type":"ContainerDied","Data":"5d331f95be2b66fbb93ad76e2e722bb45bd8f59dd246ba3c99f44a63ac3991bc"} Oct 31 16:26:20.270957 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.270533 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz" Oct 31 16:26:20.502611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.489158 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz"] Oct 31 16:26:20.509223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.503292 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:20.554813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.554781 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:26:20.564716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.564684 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f27146d0-6656c5865b-jp6tz"] Oct 31 16:26:20.648542 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.648506 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-metrics-certs\") pod \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " Oct 31 16:26:20.649863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.649836 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7baf31b-91da-426e-adc2-ed181c9bfcc8-kube-api-access-c2jxt\" (UniqueName: \"kubernetes.io/projected/b7baf31b-91da-426e-adc2-ed181c9bfcc8-kube-api-access-c2jxt\") pod \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " Oct 31 16:26:20.650434 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.650098 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7baf31b-91da-426e-adc2-ed181c9bfcc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7baf31b-91da-426e-adc2-ed181c9bfcc8-openshift-service-ca\") pod \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " Oct 31 16:26:20.650709 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.650688 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls\") pod \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\" (UID: \"b7baf31b-91da-426e-adc2-ed181c9bfcc8\") " Oct 31 16:26:20.657210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.656446 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7baf31b-91da-426e-adc2-ed181c9bfcc8-openshift-service-ca" pod "b7baf31b-91da-426e-adc2-ed181c9bfcc8" (UID: "b7baf31b-91da-426e-adc2-ed181c9bfcc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:20.670201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.668613 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls" pod "b7baf31b-91da-426e-adc2-ed181c9bfcc8" (UID: "b7baf31b-91da-426e-adc2-ed181c9bfcc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:20.670201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.668617 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-metrics-certs" pod "b7baf31b-91da-426e-adc2-ed181c9bfcc8" (UID: "b7baf31b-91da-426e-adc2-ed181c9bfcc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:20.670201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.669255 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7baf31b-91da-426e-adc2-ed181c9bfcc8-kube-api-access-c2jxt" pod "b7baf31b-91da-426e-adc2-ed181c9bfcc8" (UID: "b7baf31b-91da-426e-adc2-ed181c9bfcc8"). InnerVolumeSpecName "kube-api-access-c2jxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:20.753390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.753299 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2jxt\" (UniqueName: \"kubernetes.io/projected/b7baf31b-91da-426e-adc2-ed181c9bfcc8-kube-api-access-c2jxt\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:20.753390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.753331 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7baf31b-91da-426e-adc2-ed181c9bfcc8-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:20.753390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.753346 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:20.753390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.753357 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7baf31b-91da-426e-adc2-ed181c9bfcc8-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:20.764547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:20.764507 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:21.092496 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.092405 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd10a17-7353-482d-b9d6-be689c4100d9" path="/var/lib/kubelet/pods/3cd10a17-7353-482d-b9d6-be689c4100d9/volumes" Oct 31 16:26:21.228621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.219661 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp"] Oct 31 16:26:21.228621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.219906 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" podUID="2e96e552-44b8-4238-ae01-1a6002cf83aa" containerName="maas-api" containerID="cri-o://4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710" gracePeriod=30 Oct 31 16:26:21.276247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.253608 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 16:26:21.276247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.253669 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 16:26:21.276247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.255274 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:21.289856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.284386 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" event={"ID":"b7baf31b-91da-426e-adc2-ed181c9bfcc8","Type":"ContainerDied","Data":"b76e5ed333c354c1f443599cd5f55198c65db8eb399fff6d3a46f20e84ea5c45"} Oct 31 16:26:21.289856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.284471 4880 scope.go:122] "RemoveContainer" containerID="5d331f95be2b66fbb93ad76e2e722bb45bd8f59dd246ba3c99f44a63ac3991bc" Oct 31 16:26:21.289856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.284656 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg" Oct 31 16:26:21.541186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.531183 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg"] Oct 31 16:26:21.541186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.536638 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a08562e5-5ffd54c4c7-5mwmg"] Oct 31 16:26:21.601654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.599586 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:21.864188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:21.863305 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:22.029624 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.029441 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:22.112969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.112944 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:26:22.174416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.174336 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:22.211673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.211638 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-maas-api-tls\") pod \"2e96e552-44b8-4238-ae01-1a6002cf83aa\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " Oct 31 16:26:22.211864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.211698 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-metrics-certs\") pod \"2e96e552-44b8-4238-ae01-1a6002cf83aa\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " Oct 31 16:26:22.211864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.211721 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e96e552-44b8-4238-ae01-1a6002cf83aa-kube-api-access-jfql4\" (UniqueName: \"kubernetes.io/projected/2e96e552-44b8-4238-ae01-1a6002cf83aa-kube-api-access-jfql4\") pod \"2e96e552-44b8-4238-ae01-1a6002cf83aa\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " Oct 31 16:26:22.211992 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.211974 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e96e552-44b8-4238-ae01-1a6002cf83aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e96e552-44b8-4238-ae01-1a6002cf83aa-openshift-service-ca\") pod \"2e96e552-44b8-4238-ae01-1a6002cf83aa\" (UID: \"2e96e552-44b8-4238-ae01-1a6002cf83aa\") " Oct 31 16:26:22.212908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.212522 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e96e552-44b8-4238-ae01-1a6002cf83aa-openshift-service-ca" pod "2e96e552-44b8-4238-ae01-1a6002cf83aa" (UID: "2e96e552-44b8-4238-ae01-1a6002cf83aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:22.212908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.212823 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e96e552-44b8-4238-ae01-1a6002cf83aa-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:22.214509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.214467 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-metrics-certs" pod "2e96e552-44b8-4238-ae01-1a6002cf83aa" (UID: "2e96e552-44b8-4238-ae01-1a6002cf83aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:22.214620 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.214540 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-maas-api-tls" pod "2e96e552-44b8-4238-ae01-1a6002cf83aa" (UID: "2e96e552-44b8-4238-ae01-1a6002cf83aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:26:22.215945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.215923 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e96e552-44b8-4238-ae01-1a6002cf83aa-kube-api-access-jfql4" pod "2e96e552-44b8-4238-ae01-1a6002cf83aa" (UID: "2e96e552-44b8-4238-ae01-1a6002cf83aa"). InnerVolumeSpecName "kube-api-access-jfql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:22.241163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.232240 4880 patch_prober.go:29] interesting pod/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 16:26:22.241163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.232292 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 16:26:22.319984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.319818 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:22.319984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.319852 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e96e552-44b8-4238-ae01-1a6002cf83aa-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:22.319984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.319866 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfql4\" (UniqueName: \"kubernetes.io/projected/2e96e552-44b8-4238-ae01-1a6002cf83aa-kube-api-access-jfql4\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:22.377159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.374026 4880 generic.go:356] "Generic (PLEG): container finished" podID="2e96e552-44b8-4238-ae01-1a6002cf83aa" containerID="4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710" exitCode=0 Oct 31 16:26:22.377159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.374285 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" event={"ID":"2e96e552-44b8-4238-ae01-1a6002cf83aa","Type":"ContainerDied","Data":"4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710"} Oct 31 16:26:22.377159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.374353 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" event={"ID":"2e96e552-44b8-4238-ae01-1a6002cf83aa","Type":"ContainerDied","Data":"ee0e25141bc2504a51e40fe9990eec0b1ac9a55bf55dc296d210e20caff4d1fe"} Oct 31 16:26:22.377159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.374400 4880 scope.go:122] "RemoveContainer" containerID="4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710" Oct 31 16:26:22.377159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.374572 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp" Oct 31 16:26:22.408838 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.408536 4880 scope.go:122] "RemoveContainer" containerID="4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710" Oct 31 16:26:22.409269 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:26:22.409120 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710\": container with ID starting with 4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710 not found: ID does not exist" containerID="4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710" Oct 31 16:26:22.409487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.409412 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710"} err="failed to get container status \"4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710\": rpc error: code = NotFound desc = could not find container \"4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710\": container with ID starting with 4f1d5deff7dbdd985aa0be8016982edaeb90f23dcd566a07d09b72b45d634710 not found: ID does not exist" Oct 31 16:26:22.635287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.628788 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:22.674789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.674756 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp"] Oct 31 16:26:22.697482 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:22.697443 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-caa664e1-6cfb77798c-jzxbp"] Oct 31 16:26:23.096789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.096710 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e96e552-44b8-4238-ae01-1a6002cf83aa" path="/var/lib/kubelet/pods/2e96e552-44b8-4238-ae01-1a6002cf83aa/volumes" Oct 31 16:26:23.097462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.097440 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7baf31b-91da-426e-adc2-ed181c9bfcc8" path="/var/lib/kubelet/pods/b7baf31b-91da-426e-adc2-ed181c9bfcc8/volumes" Oct 31 16:26:23.225104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.216006 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:23.237174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.234507 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc"] Oct 31 16:26:23.246659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246628 4880 cpu_manager.go:414] "containerMap: removing container" podUID="2e96e552-44b8-4238-ae01-1a6002cf83aa" containerName="maas-api" Oct 31 16:26:23.246659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246658 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e96e552-44b8-4238-ae01-1a6002cf83aa" containerName="maas-api" Oct 31 16:26:23.246886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246700 4880 cpu_manager.go:414] "containerMap: removing container" podUID="3cd10a17-7353-482d-b9d6-be689c4100d9" containerName="maas-api" Oct 31 16:26:23.246886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246709 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd10a17-7353-482d-b9d6-be689c4100d9" containerName="maas-api" Oct 31 16:26:23.246886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246738 4880 cpu_manager.go:414] "containerMap: removing container" podUID="b7baf31b-91da-426e-adc2-ed181c9bfcc8" containerName="maas-api" Oct 31 16:26:23.246886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246747 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7baf31b-91da-426e-adc2-ed181c9bfcc8" containerName="maas-api" Oct 31 16:26:23.247073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246910 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e96e552-44b8-4238-ae01-1a6002cf83aa" containerName="maas-api" Oct 31 16:26:23.247073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246926 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd10a17-7353-482d-b9d6-be689c4100d9" containerName="maas-api" Oct 31 16:26:23.247073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.246939 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7baf31b-91da-426e-adc2-ed181c9bfcc8" containerName="maas-api" Oct 31 16:26:23.259613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.259386 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.317481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.314910 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1bb9544-openshift-default-dockercfg-nxw5j\"" Oct 31 16:26:23.333097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.333064 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc"] Oct 31 16:26:23.404422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404384 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-data\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404438 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-credential-socket\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404493 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-podinfo\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404603 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-envoy\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404659 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhnn\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-kube-api-access-qzhnn\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404728 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istiod-ca-cert\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404770 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-certs\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404886 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-socket\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.404941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.404923 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-token\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.423841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.422373 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:23.423841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.423662 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:23.427316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.427276 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:23.439161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.438076 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:23.507838 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.507800 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istiod-ca-cert\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.507999 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.507860 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-certs\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508001 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-socket\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508035 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-token\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508128 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-data\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508225 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508182 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-credential-socket\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508252 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-podinfo\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508384 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-envoy\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508448 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhnn\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-kube-api-access-qzhnn\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508490 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-certs\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508576 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-socket\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.508864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508656 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istiod-ca-cert\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.509293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.508960 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-data\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.510578 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.510555 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-credential-socket\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.511205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.511180 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-envoy\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.511291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.511230 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-podinfo\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.561993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.561958 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-token\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.563049 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.563019 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhnn\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-kube-api-access-qzhnn\") pod \"e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.586590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.586557 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:23.636332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.636301 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:23.745973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.745938 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:23.865801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.865767 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l"] Oct 31 16:26:23.900096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.896745 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr"] Oct 31 16:26:23.900096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.897022 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" containerID="cri-o://a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41" gracePeriod=30 Oct 31 16:26:23.911631 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:23.911438 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:24.138624 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.138590 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc"] Oct 31 16:26:24.142650 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:26:24.142620 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfdc58e_fcd7_4f39_9723_ec160c7bf37f.slice/crio-4c68d2702379bc4d50736c91a64a3732dad4a7ddda66e5bd494c363471ccf466 WatchSource:0}: Error finding container 4c68d2702379bc4d50736c91a64a3732dad4a7ddda66e5bd494c363471ccf466: Status 404 returned error can't find the container with id 4c68d2702379bc4d50736c91a64a3732dad4a7ddda66e5bd494c363471ccf466 Oct 31 16:26:24.145738 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.145706 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:24.145834 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.145788 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:24.145891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.145836 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:24.467217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.467046 4880 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" secret="" err="secret \"e2e-derive-caa664e1-openshift-default-dockercfg-cbxhc\" not found" Oct 31 16:26:24.468689 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.468658 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" event={"ID":"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f","Type":"ContainerStarted","Data":"bbb854bd347b3eabc3c852fed37f1e181ec44111125a77967c67749e2babf4f4"} Oct 31 16:26:24.468833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.468715 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" event={"ID":"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f","Type":"ContainerStarted","Data":"4c68d2702379bc4d50736c91a64a3732dad4a7ddda66e5bd494c363471ccf466"} Oct 31 16:26:24.580200 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.577191 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podStartSLOduration=1.57717244 podStartE2EDuration="1.57717244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:26:24.576538341 +0000 UTC m=+1431.611833146" watchObservedRunningTime="2026-10-31 16:26:24.57717244 +0000 UTC m=+1431.612467246" Oct 31 16:26:24.636943 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.636910 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:24.856920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:24.856830 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:25.017158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:25.016899 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:25.171167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:25.169527 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:25.478413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:25.478280 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" containerID="cri-o://b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76" gracePeriod=30 Oct 31 16:26:25.643983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:25.643944 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:26.220105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:26.220063 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:26.462915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:26.462881 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:26.489571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:26.489397 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:26.490851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:26.490832 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:26:26.679048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:26.679013 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:26.843607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:26.843512 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:27.004304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:27.004266 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:28.069282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:28.069243 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:28.904651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:28.904607 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:29.205276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.205257 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:26:29.261304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261277 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-istio-token\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-istio-token\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261331 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-credential-socket\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261482 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261433 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-socket\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261509 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-envoy\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261548 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-data\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261581 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8bc1761-7028-4947-b122-ed7dbc238736-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8bc1761-7028-4947-b122-ed7dbc238736-istio-podinfo\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261621 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-credential-socket" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:29.261683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261670 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-certs\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261692 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-socket" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:29.261798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261751 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-kube-api-access-t2pqk\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-kube-api-access-t2pqk\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.261798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261774 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-data" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:29.261955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261831 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-certs" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:29.261955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.261847 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bc1761-7028-4947-b122-ed7dbc238736-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8bc1761-7028-4947-b122-ed7dbc238736-istiod-ca-cert\") pod \"d8bc1761-7028-4947-b122-ed7dbc238736\" (UID: \"d8bc1761-7028-4947-b122-ed7dbc238736\") " Oct 31 16:26:29.262248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.262224 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bc1761-7028-4947-b122-ed7dbc238736-istiod-ca-cert" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:29.262642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.262621 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.262705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.262649 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.262705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.262663 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.262705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.262675 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8bc1761-7028-4947-b122-ed7dbc238736-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.262705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.262688 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.263656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.263626 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-envoy" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:29.263758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.263690 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-istio-token" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:29.264043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.264026 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8bc1761-7028-4947-b122-ed7dbc238736-istio-podinfo" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:26:29.264107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.264075 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-kube-api-access-t2pqk" pod "d8bc1761-7028-4947-b122-ed7dbc238736" (UID: "d8bc1761-7028-4947-b122-ed7dbc238736"). InnerVolumeSpecName "kube-api-access-t2pqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:29.364835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.364803 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2pqk\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-kube-api-access-t2pqk\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.364835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.364835 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8bc1761-7028-4947-b122-ed7dbc238736-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.364835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.364849 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8bc1761-7028-4947-b122-ed7dbc238736-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.365096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.364863 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8bc1761-7028-4947-b122-ed7dbc238736-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:29.519730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.519648 4880 generic.go:356] "Generic (PLEG): container finished" podID="d8bc1761-7028-4947-b122-ed7dbc238736" containerID="a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41" exitCode=0 Oct 31 16:26:29.519730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.519698 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" Oct 31 16:26:29.519985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.519730 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" event={"ID":"d8bc1761-7028-4947-b122-ed7dbc238736","Type":"ContainerDied","Data":"a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41"} Oct 31 16:26:29.519985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.519791 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr" event={"ID":"d8bc1761-7028-4947-b122-ed7dbc238736","Type":"ContainerDied","Data":"a84c849ce15e211a53709c789619daf1993e3841c0dbd5a8be8e480eda90bd10"} Oct 31 16:26:29.519985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.519822 4880 scope.go:122] "RemoveContainer" containerID="a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41" Oct 31 16:26:29.533040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.533021 4880 scope.go:122] "RemoveContainer" containerID="a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41" Oct 31 16:26:29.533379 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:26:29.533360 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41\": container with ID starting with a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41 not found: ID does not exist" containerID="a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41" Oct 31 16:26:29.533450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.533391 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41"} err="failed to get container status \"a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41\": rpc error: code = NotFound desc = could not find container \"a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41\": container with ID starting with a3458f1b6a078f6497edae84846fba37655fd62cff10ef29a7ae7530f9749a41 not found: ID does not exist" Oct 31 16:26:29.672988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.672956 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr"] Oct 31 16:26:29.682219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.682187 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-npbdr"] Oct 31 16:26:29.819250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:29.819160 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:30.012587 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.012554 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:30.662096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.662054 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:30.818841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.818812 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:30.867157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.866332 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:30.909760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.909730 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-podinfo\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.909940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.909784 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-token\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-token\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.909940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.909813 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-socket\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.909940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.909921 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-certs\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.910166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910008 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-credential-socket\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.910166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910073 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-envoy\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.910166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910077 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-socket" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:30.910314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910191 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-certs" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:30.910314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910247 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-data\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.910314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910282 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-kube-api-access-k7c9q\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-kube-api-access-k7c9q\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.910455 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910311 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-credential-socket" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:30.910455 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910360 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istiod-ca-cert\") pod \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\" (UID: \"f63a05d2-61b8-4a02-89bf-9146716c9d5b\") " Oct 31 16:26:30.911236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910449 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-data" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:30.911236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.910758 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istiod-ca-cert" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:30.911336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.911313 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:30.911391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.911333 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:30.911391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.911348 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:30.911391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.911363 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:30.911391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.911377 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:30.912196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.912170 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-token" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:30.912401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.912373 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-envoy" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:30.912401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.912392 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-podinfo" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:26:30.912580 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:30.912562 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-kube-api-access-k7c9q" pod "f63a05d2-61b8-4a02-89bf-9146716c9d5b" (UID: "f63a05d2-61b8-4a02-89bf-9146716c9d5b"). InnerVolumeSpecName "kube-api-access-k7c9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:31.012818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.012778 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:31.012818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.012812 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7c9q\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-kube-api-access-k7c9q\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:31.013067 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.012829 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:31.013067 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.012843 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63a05d2-61b8-4a02-89bf-9146716c9d5b-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:31.092359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.092287 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" path="/var/lib/kubelet/pods/d8bc1761-7028-4947-b122-ed7dbc238736/volumes" Oct 31 16:26:31.157220 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.157180 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:31.540315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.540279 4880 generic.go:356] "Generic (PLEG): container finished" podID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerID="b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76" exitCode=0 Oct 31 16:26:31.540457 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.540332 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" event={"ID":"f63a05d2-61b8-4a02-89bf-9146716c9d5b","Type":"ContainerDied","Data":"b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76"} Oct 31 16:26:31.540457 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.540349 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" Oct 31 16:26:31.540457 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.540412 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l" event={"ID":"f63a05d2-61b8-4a02-89bf-9146716c9d5b","Type":"ContainerDied","Data":"7ed525bc1cbe22f797b19de88921d5f9756f556350554f8c4814092117026921"} Oct 31 16:26:31.540457 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.540441 4880 scope.go:122] "RemoveContainer" containerID="b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76" Oct 31 16:26:31.551086 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.551067 4880 scope.go:122] "RemoveContainer" containerID="b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76" Oct 31 16:26:31.551391 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:26:31.551368 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76\": container with ID starting with b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76 not found: ID does not exist" containerID="b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76" Oct 31 16:26:31.551460 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.551403 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76"} err="failed to get container status \"b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76\": rpc error: code = NotFound desc = could not find container \"b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76\": container with ID starting with b482de944c86a3ce567e15da8c42988f53daa95a1ed74765b751e84fdc124e76 not found: ID does not exist" Oct 31 16:26:31.740507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.740474 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l"] Oct 31 16:26:31.786215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:31.786183 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-caa664e1-openshift-default-bd4767cd9-tq46l"] Oct 31 16:26:33.093232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:33.093201 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" path="/var/lib/kubelet/pods/f63a05d2-61b8-4a02-89bf-9146716c9d5b/volumes" Oct 31 16:26:33.880658 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:26:33.880570 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:26:34.173094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:34.173058 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:36.764012 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:36.763975 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:39.055113 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:39.055064 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:39.929011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:39.928969 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:40.073181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:40.073122 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:43.856039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:43.856004 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:47.244250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:47.244216 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:26:47.244952 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:47.244921 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:26:48.737117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:48.737079 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:52.987619 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:52.987583 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf"] Oct 31 16:26:52.988087 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:52.987886 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" containerID="cri-o://86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e" gracePeriod=30 Oct 31 16:26:53.507206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.507172 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:53.803685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.803607 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq"] Oct 31 16:26:53.804270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.804253 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" Oct 31 16:26:53.804270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.804269 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" Oct 31 16:26:53.804426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.804289 4880 cpu_manager.go:414] "containerMap: removing container" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" Oct 31 16:26:53.804426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.804294 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" Oct 31 16:26:53.804426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.804394 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63a05d2-61b8-4a02-89bf-9146716c9d5b" containerName="istio-proxy" Oct 31 16:26:53.804426 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.804404 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bc1761-7028-4947-b122-ed7dbc238736" containerName="istio-proxy" Oct 31 16:26:53.808192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.808172 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.857090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.857064 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2bg6m\"" Oct 31 16:26:53.857215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.857090 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 16:26:53.857215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.857107 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 16:26:53.857215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.857116 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 16:26:53.908820 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.908790 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.908969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.908950 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.909032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.908983 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62919192-2a24-4d1f-a67c-9d68b72f4e55-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.909032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.909009 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7q9g\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-kube-api-access-g7q9g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.909146 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.909124 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.909272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.909255 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.909361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.909345 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:53.910250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:53.910228 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq"] Oct 31 16:26:54.010827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.010796 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62919192-2a24-4d1f-a67c-9d68b72f4e55-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.011312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.010835 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q9g\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-kube-api-access-g7q9g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.011312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.010899 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.011312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.011060 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.011312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.011145 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.011312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.011242 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.011570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.011366 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.011873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.011844 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.013473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.013445 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62919192-2a24-4d1f-a67c-9d68b72f4e55-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.013579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.013450 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.013579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.013494 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.013656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.013586 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.063530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.063475 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7q9g\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-kube-api-access-g7q9g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.063530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.063496 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t9zcq\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.120859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.120836 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:54.547853 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.547821 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq"] Oct 31 16:26:54.552113 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:26:54.552089 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62919192_2a24_4d1f_a67c_9d68b72f4e55.slice/crio-b52dc8f301b131adc5222c6c90ca03a4befca2018e1a0d4dc99bbdbf1115579c WatchSource:0}: Error finding container b52dc8f301b131adc5222c6c90ca03a4befca2018e1a0d4dc99bbdbf1115579c: Status 404 returned error can't find the container with id b52dc8f301b131adc5222c6c90ca03a4befca2018e1a0d4dc99bbdbf1115579c Oct 31 16:26:54.772803 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:54.772767 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" event={"ID":"62919192-2a24-4d1f-a67c-9d68b72f4e55","Type":"ContainerStarted","Data":"b52dc8f301b131adc5222c6c90ca03a4befca2018e1a0d4dc99bbdbf1115579c"} Oct 31 16:26:57.975938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:57.975901 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:26:57.976433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:57.975969 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:26:58.284623 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.284599 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:26:58.342009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.341974 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:26:58.375185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.375114 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-istio-token\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-istio-token\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.375369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.375246 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a52591-07b2-491e-a873-a537c7278595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a52591-07b2-491e-a873-a537c7278595-istiod-ca-cert\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.375369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.375300 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5a52591-07b2-491e-a873-a537c7278595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a52591-07b2-491e-a873-a537c7278595-istio-podinfo\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.375369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.375333 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-data\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.375613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.375377 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-credential-socket\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.375613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.375445 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-envoy\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.375938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.375902 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-certs\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.378222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.376196 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-socket\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.378222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.376767 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-kube-api-access-p2jq6\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-kube-api-access-p2jq6\") pod \"b5a52591-07b2-491e-a873-a537c7278595\" (UID: \"b5a52591-07b2-491e-a873-a537c7278595\") " Oct 31 16:26:58.378222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.377916 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a52591-07b2-491e-a873-a537c7278595-istiod-ca-cert" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:26:58.385774 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.385726 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-credential-socket" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:58.386082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.386044 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-data" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:58.387243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.387116 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-socket" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:58.387243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.387206 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-certs" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:58.405179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.405095 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a52591-07b2-491e-a873-a537c7278595-istio-podinfo" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:26:58.407546 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.407494 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-envoy" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:26:58.408281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.408019 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-kube-api-access-p2jq6" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "kube-api-access-p2jq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:58.410562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.410501 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-istio-token" pod "b5a52591-07b2-491e-a873-a537c7278595" (UID: "b5a52591-07b2-491e-a873-a537c7278595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:26:58.479552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479517 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jq6\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-kube-api-access-p2jq6\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.479552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479550 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a52591-07b2-491e-a873-a537c7278595-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.480385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479562 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a52591-07b2-491e-a873-a537c7278595-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.480385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479576 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a52591-07b2-491e-a873-a537c7278595-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.480385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479588 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.480385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479599 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.480385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479610 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.480385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479622 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.480385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.479634 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a52591-07b2-491e-a873-a537c7278595-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:26:58.750969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.750940 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:58.751048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.751002 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:26:58.805234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.805208 4880 generic.go:356] "Generic (PLEG): container finished" podID="b5a52591-07b2-491e-a873-a537c7278595" containerID="86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e" exitCode=0 Oct 31 16:26:58.805335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.805254 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" Oct 31 16:26:58.805335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.805262 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" event={"ID":"b5a52591-07b2-491e-a873-a537c7278595","Type":"ContainerDied","Data":"86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e"} Oct 31 16:26:58.805335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.805309 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf" event={"ID":"b5a52591-07b2-491e-a873-a537c7278595","Type":"ContainerDied","Data":"ae8c474d082a5d5849804e39f8635a8c601f4c062acd8b10001b0e30f3d14a6b"} Oct 31 16:26:58.805461 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.805334 4880 scope.go:122] "RemoveContainer" containerID="86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e" Oct 31 16:26:58.815790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.815773 4880 scope.go:122] "RemoveContainer" containerID="86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e" Oct 31 16:26:58.816086 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:26:58.816065 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e\": container with ID starting with 86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e not found: ID does not exist" containerID="86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e" Oct 31 16:26:58.816228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.816097 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e"} err="failed to get container status \"86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e\": rpc error: code = NotFound desc = could not find container \"86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e\": container with ID starting with 86a462220bc428404ab3e31b52e3ba87d296f5730f3f8c8ff6abb94239ac505e not found: ID does not exist" Oct 31 16:26:58.961499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.961467 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf"] Oct 31 16:26:58.966707 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:58.966683 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-lnzgf"] Oct 31 16:26:59.091793 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:59.091722 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a52591-07b2-491e-a873-a537c7278595" path="/var/lib/kubelet/pods/b5a52591-07b2-491e-a873-a537c7278595/volumes" Oct 31 16:26:59.816074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:59.816039 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" event={"ID":"62919192-2a24-4d1f-a67c-9d68b72f4e55","Type":"ContainerStarted","Data":"fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66"} Oct 31 16:26:59.816286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:59.816249 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:26:59.817858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:59.817833 4880 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t9zcq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:26:59.817968 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:26:59.817888 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" podUID="62919192-2a24-4d1f-a67c-9d68b72f4e55" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:27:00.010825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:00.010769 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" podStartSLOduration=2.814981872 podStartE2EDuration="7.010753822s" totalImagesPullingTime="4.19577195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:00.009211347 +0000 UTC m=+1467.044506152" watchObservedRunningTime="2026-10-31 16:27:00.010753822 +0000 UTC m=+1467.046048635" Oct 31 16:27:00.825110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:00.825086 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:27:02.429957 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:02.429924 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:03.169354 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:03.169321 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:03.882058 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:27:03.882016 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:27:07.670932 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:07.670892 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:07.909281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:07.909215 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:08.055307 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:08.055176 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:08.213629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:08.213594 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:08.389953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:08.389922 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:08.542613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:08.542578 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:08.748397 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:08.748308 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:12.968683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:12.968594 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:13.126568 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:13.126057 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:13.269356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:13.269283 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:14.154672 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:14.154635 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:14.454072 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:14.453980 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:17.979536 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:17.979474 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:18.166627 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:18.166586 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:18.319373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:18.319292 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:18.468218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:18.468185 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:19.027037 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:19.027003 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:19.456425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:19.456389 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:23.145216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:23.145180 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:23.341183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:23.341118 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:23.498557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:23.498467 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:23.664214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:23.664174 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:26.289944 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:26.289900 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:26.465114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:26.465079 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:28.664639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:28.664601 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:31.450207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:31.450166 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:33.317907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.317877 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:27:33.321292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.321268 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:27:33.326417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.326397 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:27:33.329513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.329491 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:27:33.337673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.337639 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:27:33.340751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.340731 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:27:33.543860 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.543825 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:33.703666 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:33.703628 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:33.881368 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:27:33.881331 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:27:34.275713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:34.275606 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:34.538224 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:34.537938 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:38.447161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:38.447105 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:38.606895 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:38.606861 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:38.765366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:38.765281 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:41.632058 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:41.632004 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:41.796002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:41.790078 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:42.013456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.013418 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f"] Oct 31 16:27:42.014597 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.014570 4880 cpu_manager.go:414] "containerMap: removing container" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" Oct 31 16:27:42.014597 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.014597 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" Oct 31 16:27:42.014789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.014777 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a52591-07b2-491e-a873-a537c7278595" containerName="istio-proxy" Oct 31 16:27:42.019159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.019122 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.068563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.068524 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f"] Oct 31 16:27:42.108865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.108765 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dc67dd\"" Oct 31 16:27:42.109108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.108939 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dc67dd-dockercfg-d554n\"" Oct 31 16:27:42.132270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.125029 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwth\" (UniqueName: \"kubernetes.io/projected/73696f60-ddbd-43cf-9972-77497f586709-kube-api-access-vzwth\") pod \"payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.132270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.125168 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73696f60-ddbd-43cf-9972-77497f586709-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.226254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.226215 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwth\" (UniqueName: \"kubernetes.io/projected/73696f60-ddbd-43cf-9972-77497f586709-kube-api-access-vzwth\") pod \"payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.226498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.226352 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73696f60-ddbd-43cf-9972-77497f586709-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.227262 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.227241 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73696f60-ddbd-43cf-9972-77497f586709-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.279440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.279357 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwth\" (UniqueName: \"kubernetes.io/projected/73696f60-ddbd-43cf-9972-77497f586709-kube-api-access-vzwth\") pod \"payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.336683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.336650 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:42.793402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:42.793364 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f"] Oct 31 16:27:42.798311 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:27:42.798282 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73696f60_ddbd_43cf_9972_77497f586709.slice/crio-59aed8864a6f2e78cbc3995eb6de00061426f3dbf3e3a227defb58387e15cd8d WatchSource:0}: Error finding container 59aed8864a6f2e78cbc3995eb6de00061426f3dbf3e3a227defb58387e15cd8d: Status 404 returned error can't find the container with id 59aed8864a6f2e78cbc3995eb6de00061426f3dbf3e3a227defb58387e15cd8d Oct 31 16:27:43.178599 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:43.178560 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" event={"ID":"73696f60-ddbd-43cf-9972-77497f586709","Type":"ContainerStarted","Data":"5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9"} Oct 31 16:27:43.178766 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:43.178622 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" event={"ID":"73696f60-ddbd-43cf-9972-77497f586709","Type":"ContainerStarted","Data":"59aed8864a6f2e78cbc3995eb6de00061426f3dbf3e3a227defb58387e15cd8d"} Oct 31 16:27:43.178766 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:43.178650 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:43.285429 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:43.285380 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" podStartSLOduration=2.285364319 podStartE2EDuration="2.285364319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:27:43.283129661 +0000 UTC m=+1510.318424466" watchObservedRunningTime="2026-10-31 16:27:43.285364319 +0000 UTC m=+1510.320659122" Oct 31 16:27:43.741475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:43.741439 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:44.717770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:44.717719 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:45.888984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:45.888953 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:46.079913 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:46.079873 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:48.596572 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:48.596531 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:50.670029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:50.669991 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:53.410733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:53.410704 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:27:54.189191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:54.189152 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:27:58.306201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:27:58.306162 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:02.799230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.799155 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx"] Oct 31 16:28:02.816456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.816430 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:02.855520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.855493 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx"] Oct 31 16:28:02.865769 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.865744 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-d04acd-serving-cert\"" Oct 31 16:28:02.903082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.903054 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:02.903259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.903196 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79633c9-3f3e-4d00-9e7f-445219ca919d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:02.903361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.903342 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrvx\" (UniqueName: \"kubernetes.io/projected/b79633c9-3f3e-4d00-9e7f-445219ca919d-kube-api-access-7lrvx\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:02.903509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:02.903492 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.005126 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.005096 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrvx\" (UniqueName: \"kubernetes.io/projected/b79633c9-3f3e-4d00-9e7f-445219ca919d-kube-api-access-7lrvx\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.005254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.005238 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.005355 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.005342 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.005417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.005389 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79633c9-3f3e-4d00-9e7f-445219ca919d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.005501 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:03.005481 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-serving-cert: secret "maas-api-e2e-shared-a-w5-d04acd-serving-cert" not found Oct 31 16:28:03.005571 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:03.005560 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls podName:b79633c9-3f3e-4d00-9e7f-445219ca919d nodeName:}" failed. No retries permitted until 2026-10-31 16:28:03.505542721 +0000 UTC m=+1530.540837521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls") pod "maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" (UID: "b79633c9-3f3e-4d00-9e7f-445219ca919d") : secret "maas-api-e2e-shared-a-w5-d04acd-serving-cert" not found Oct 31 16:28:03.005959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.005943 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79633c9-3f3e-4d00-9e7f-445219ca919d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.007514 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.007489 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.012414 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.012389 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:03.062852 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.062771 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrvx\" (UniqueName: \"kubernetes.io/projected/b79633c9-3f3e-4d00-9e7f-445219ca919d-kube-api-access-7lrvx\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.168210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.168171 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:03.369027 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.368927 4880 generic.go:356] "Generic (PLEG): container finished" podID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerID="bbb854bd347b3eabc3c852fed37f1e181ec44111125a77967c67749e2babf4f4" exitCode=0 Oct 31 16:28:03.369464 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.369167 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" event={"ID":"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f","Type":"ContainerDied","Data":"bbb854bd347b3eabc3c852fed37f1e181ec44111125a77967c67749e2babf4f4"} Oct 31 16:28:03.369681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.369652 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:03.370328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.370299 4880 scope.go:122] "RemoveContainer" containerID="bbb854bd347b3eabc3c852fed37f1e181ec44111125a77967c67749e2babf4f4" Oct 31 16:28:03.373213 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.373110 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:28:03.373477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.373411 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:28:03.373577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.373534 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:28:03.514836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.514400 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.519273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.519241 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.748067 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:03.748034 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:03.893586 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:03.893549 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:28:04.148168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:04.148107 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx"] Oct 31 16:28:04.153994 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:28:04.153954 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79633c9_3f3e_4d00_9e7f_445219ca919d.slice/crio-5e453fb790d5102e5cdb3720c9ab25ab076279a31948beae4c60acf6a721cbe1 WatchSource:0}: Error finding container 5e453fb790d5102e5cdb3720c9ab25ab076279a31948beae4c60acf6a721cbe1: Status 404 returned error can't find the container with id 5e453fb790d5102e5cdb3720c9ab25ab076279a31948beae4c60acf6a721cbe1 Oct 31 16:28:04.378598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:04.378559 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" event={"ID":"b79633c9-3f3e-4d00-9e7f-445219ca919d","Type":"ContainerStarted","Data":"5e453fb790d5102e5cdb3720c9ab25ab076279a31948beae4c60acf6a721cbe1"} Oct 31 16:28:04.380665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:04.380634 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" event={"ID":"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f","Type":"ContainerStarted","Data":"ac9dc4f6493df4a93cb874b80672cc63a01c2e3383155b8a7b38f73f9f53188a"} Oct 31 16:28:04.644412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:04.641232 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:28:04.644412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:04.641359 4880 patch_prober.go:29] interesting pod/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 16:28:04.644412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:04.641402 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 16:28:05.395621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:05.395575 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" event={"ID":"b79633c9-3f3e-4d00-9e7f-445219ca919d","Type":"ContainerStarted","Data":"adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5"} Oct 31 16:28:05.396046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:05.395781 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:05.504345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:05.504287 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" podStartSLOduration=2.553246622 podStartE2EDuration="3.504270075s" totalImagesPullingTime="951.023453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:05.502474036 +0000 UTC m=+1532.537768853" watchObservedRunningTime="2026-10-31 16:28:05.504270075 +0000 UTC m=+1532.539564883" Oct 31 16:28:05.542162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:05.541830 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:05.660250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:05.649983 4880 patch_prober.go:29] interesting pod/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 16:28:05.660250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:05.650044 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 16:28:06.645564 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:06.645528 4880 patch_prober.go:29] interesting pod/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 16:28:06.645983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:06.645588 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 16:28:07.640491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:07.640452 4880 patch_prober.go:29] interesting pod/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 16:28:07.640749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:07.640723 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 16:28:08.482815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:08.482781 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:08.642164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:08.641901 4880 patch_prober.go:29] interesting pod/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 31 16:28:08.642164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:08.641974 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 31 16:28:09.642770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:09.642740 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:28:09.643233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:09.643026 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:28:09.643739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:09.643721 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:28:11.409747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:11.409717 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:28:13.841991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:13.841950 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:18.705481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.705448 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm"] Oct 31 16:28:18.708832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.708812 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.803860 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.803822 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a7907d70-serving-cert\"" Oct 31 16:28:18.805925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.805896 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-metrics-certs\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.806045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.805944 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9qr\" (UniqueName: \"kubernetes.io/projected/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-kube-api-access-2h9qr\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.806106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.806063 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.806412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.806388 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-openshift-service-ca\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.811084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.811063 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm"] Oct 31 16:28:18.908211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.908178 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-openshift-service-ca\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.908377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.908300 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-metrics-certs\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.908377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.908322 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9qr\" (UniqueName: \"kubernetes.io/projected/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-kube-api-access-2h9qr\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.908377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.908350 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.908556 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:18.908534 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-serving-cert: secret "maas-api-e2e-mt-a7907d70-serving-cert" not found Oct 31 16:28:18.908645 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:18.908633 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls podName:2f1fa89b-f6d9-4a37-bd00-f86759c8b649 nodeName:}" failed. No retries permitted until 2026-10-31 16:28:19.408601692 +0000 UTC m=+1546.443896488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls") pod "maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" (UID: "2f1fa89b-f6d9-4a37-bd00-f86759c8b649") : secret "maas-api-e2e-mt-a7907d70-serving-cert" not found Oct 31 16:28:18.908995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.908968 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-openshift-service-ca\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.910725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.910704 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-metrics-certs\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:18.921409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.921384 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:18.972088 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:18.971963 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9qr\" (UniqueName: \"kubernetes.io/projected/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-kube-api-access-2h9qr\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:19.110018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:19.109979 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:19.250153 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:19.250045 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:19.416352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:19.416321 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:19.418772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:19.418747 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls\") pod \"maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:19.620833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:19.620750 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:19.995173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:19.995127 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm"] Oct 31 16:28:19.998709 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:28:19.998684 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1fa89b_f6d9_4a37_bd00_f86759c8b649.slice/crio-d9ccdd07dab58402b054eb50d7ef2a366595019318b9126fbece59bf8d3728c1 WatchSource:0}: Error finding container d9ccdd07dab58402b054eb50d7ef2a366595019318b9126fbece59bf8d3728c1: Status 404 returned error can't find the container with id d9ccdd07dab58402b054eb50d7ef2a366595019318b9126fbece59bf8d3728c1 Oct 31 16:28:20.000221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:20.000203 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:28:20.509032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:20.508995 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" event={"ID":"2f1fa89b-f6d9-4a37-bd00-f86759c8b649","Type":"ContainerStarted","Data":"d9ccdd07dab58402b054eb50d7ef2a366595019318b9126fbece59bf8d3728c1"} Oct 31 16:28:21.518220 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:21.518178 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" event={"ID":"2f1fa89b-f6d9-4a37-bd00-f86759c8b649","Type":"ContainerStarted","Data":"81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b"} Oct 31 16:28:21.518643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:21.518247 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:21.626077 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:21.626025 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" podStartSLOduration=2.73607566 podStartE2EDuration="3.626013294s" totalImagesPullingTime="889.937634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:21.624050952 +0000 UTC m=+1548.659345757" watchObservedRunningTime="2026-10-31 16:28:21.626013294 +0000 UTC m=+1548.661308099" Oct 31 16:28:22.241084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:22.241042 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:22.437341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:22.437308 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:24.517201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:24.517164 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:27.583160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:27.579329 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:28:30.191890 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:30.191855 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:31.791303 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:31.791261 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:33.254048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:33.254007 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:33.896295 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:33.896212 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:28:33.995180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:33.992097 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:34.393253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:34.393216 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:35.193235 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:35.193197 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:35.387302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:35.387252 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:38.493585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:38.493546 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:38.702559 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:38.702519 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:40.276684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.276652 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654"] Oct 31 16:28:40.282590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.282567 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.330758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.330735 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-dc67dd-serving-cert\"" Oct 31 16:28:40.361953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.361913 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88gbd\" (UniqueName: \"kubernetes.io/projected/dcedb53b-0e22-45a2-86cd-145c9958cac3-kube-api-access-88gbd\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.362305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.362278 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcedb53b-0e22-45a2-86cd-145c9958cac3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.362489 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.362339 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.362489 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.362371 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.383332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.383308 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654"] Oct 31 16:28:40.463585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.463557 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88gbd\" (UniqueName: \"kubernetes.io/projected/dcedb53b-0e22-45a2-86cd-145c9958cac3-kube-api-access-88gbd\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.463758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.463652 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcedb53b-0e22-45a2-86cd-145c9958cac3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.463758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.463684 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.463758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.463703 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.463978 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:40.463901 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-serving-cert: secret "maas-api-e2e-shared-a-w2-dc67dd-serving-cert" not found Oct 31 16:28:40.464032 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:28:40.463989 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls podName:dcedb53b-0e22-45a2-86cd-145c9958cac3 nodeName:}" failed. No retries permitted until 2026-10-31 16:28:40.963967748 +0000 UTC m=+1567.999262545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls") pod "maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" (UID: "dcedb53b-0e22-45a2-86cd-145c9958cac3") : secret "maas-api-e2e-shared-a-w2-dc67dd-serving-cert" not found Oct 31 16:28:40.464292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.464272 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcedb53b-0e22-45a2-86cd-145c9958cac3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.466051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.466032 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.518786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.518757 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88gbd\" (UniqueName: \"kubernetes.io/projected/dcedb53b-0e22-45a2-86cd-145c9958cac3-kube-api-access-88gbd\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.548687 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.548630 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:40.717088 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.717051 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:40.864731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.864631 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:40.972356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.972325 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:40.974697 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:40.974670 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:41.200671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:41.200645 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:41.581321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:41.580648 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654"] Oct 31 16:28:41.585796 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:28:41.585769 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcedb53b_0e22_45a2_86cd_145c9958cac3.slice/crio-e6f06470d4fe9f9614073bc4bd06498ef86787ad119bcb8896f4c592663ce379 WatchSource:0}: Error finding container e6f06470d4fe9f9614073bc4bd06498ef86787ad119bcb8896f4c592663ce379: Status 404 returned error can't find the container with id e6f06470d4fe9f9614073bc4bd06498ef86787ad119bcb8896f4c592663ce379 Oct 31 16:28:41.819191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:41.819157 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" event={"ID":"dcedb53b-0e22-45a2-86cd-145c9958cac3","Type":"ContainerStarted","Data":"e6f06470d4fe9f9614073bc4bd06498ef86787ad119bcb8896f4c592663ce379"} Oct 31 16:28:42.829891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:42.829808 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" event={"ID":"dcedb53b-0e22-45a2-86cd-145c9958cac3","Type":"ContainerStarted","Data":"87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78"} Oct 31 16:28:42.830304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:42.829900 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:42.937084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:42.937034 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" podStartSLOduration=1.9177634879999998 podStartE2EDuration="2.937020984s" totalImagesPullingTime="1.019257496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:42.935596743 +0000 UTC m=+1569.970891550" watchObservedRunningTime="2026-10-31 16:28:42.937020984 +0000 UTC m=+1569.972315788" Oct 31 16:28:45.641879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:45.641842 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:45.829963 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:45.829931 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:48.855661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:48.855632 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:28:50.644921 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:50.644883 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:28:55.459060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:28:55.459025 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:00.261131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:00.261092 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:03.881340 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:29:03.881303 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:29:04.466816 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:04.466781 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:05.984311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:05.984273 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:06.259904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:06.259826 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:06.413543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:06.413507 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:07.980636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:07.980604 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc"] Oct 31 16:29:07.981473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:07.981437 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" containerID="cri-o://ac9dc4f6493df4a93cb874b80672cc63a01c2e3383155b8a7b38f73f9f53188a" gracePeriod=30 Oct 31 16:29:08.003361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:08.003332 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:08.765306 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:08.765257 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:09.646898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:09.646861 4880 patch_prober.go:29] interesting pod/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:29:09.647353 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:09.646931 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:29:10.400958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:10.400923 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:10.556617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:10.556578 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:11.113454 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:11.113269 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:12.734882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:12.734843 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:12.964528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:12.964491 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:13.109674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.109636 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:13.178955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.178913 4880 generic.go:356] "Generic (PLEG): container finished" podID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerID="ac9dc4f6493df4a93cb874b80672cc63a01c2e3383155b8a7b38f73f9f53188a" exitCode=0 Oct 31 16:29:13.179174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.179093 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" event={"ID":"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f","Type":"ContainerDied","Data":"ac9dc4f6493df4a93cb874b80672cc63a01c2e3383155b8a7b38f73f9f53188a"} Oct 31 16:29:13.179246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.179204 4880 scope.go:122] "RemoveContainer" containerID="bbb854bd347b3eabc3c852fed37f1e181ec44111125a77967c67749e2babf4f4" Oct 31 16:29:13.363725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.363701 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:29:13.480465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.480432 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-token\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-token\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.480655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.480554 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-kube-api-access-qzhnn\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-kube-api-access-qzhnn\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.480723 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.480672 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-podinfo\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.480797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.480777 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istiod-ca-cert\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.480869 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.480816 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-credential-socket\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.480869 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.480845 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-data\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.480972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.480961 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-socket\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.481032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481013 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-envoy\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.481088 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481041 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-certs\") pod \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\" (UID: \"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f\") " Oct 31 16:29:13.481630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481311 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-data" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:13.481630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481415 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-socket" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:13.481630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481426 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istiod-ca-cert" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:29:13.481630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481515 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-certs" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:13.481630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481609 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-credential-socket" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:13.481898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481844 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.481898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481868 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.481898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481882 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.481898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481896 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.482040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.481909 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.486315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.486243 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-podinfo" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:29:13.486315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.486247 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-envoy" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:13.486315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.486277 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-token" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:29:13.486315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.486300 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-kube-api-access-qzhnn" pod "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" (UID: "3dfdc58e-fcd7-4f39-9723-ec160c7bf37f"). InnerVolumeSpecName "kube-api-access-qzhnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:29:13.584116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.584032 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.584116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.584062 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.584116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.584073 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzhnn\" (UniqueName: \"kubernetes.io/projected/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-kube-api-access-qzhnn\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.584116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.584083 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:13.867752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:13.867676 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:14.188836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:14.188806 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" Oct 31 16:29:14.189089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:14.188826 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc" event={"ID":"3dfdc58e-fcd7-4f39-9723-ec160c7bf37f","Type":"ContainerDied","Data":"4c68d2702379bc4d50736c91a64a3732dad4a7ddda66e5bd494c363471ccf466"} Oct 31 16:29:14.189267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:14.189118 4880 scope.go:122] "RemoveContainer" containerID="ac9dc4f6493df4a93cb874b80672cc63a01c2e3383155b8a7b38f73f9f53188a" Oct 31 16:29:14.345006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:14.344964 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc"] Oct 31 16:29:14.389726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:14.389697 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1bb9544-openshift-default-6467754d77-9qhzc"] Oct 31 16:29:14.641370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:14.641338 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:15.092031 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:15.091943 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" path="/var/lib/kubelet/pods/3dfdc58e-fcd7-4f39-9723-ec160c7bf37f/volumes" Oct 31 16:29:15.608722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:15.608681 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:15.796033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:15.796000 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:15.955121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:15.955078 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:16.155558 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:16.155526 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:16.573161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:16.567483 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:16.956223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:16.956184 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:17.814650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:17.814615 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:17.965955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:17.965921 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:18.168420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:18.168383 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:19.470334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:19.470273 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:19.666838 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:19.666801 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:21.217283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:21.217244 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:21.462534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:21.462498 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:21.620535 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:21.620454 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:21.830244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:21.830207 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:22.029040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:22.028984 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:22.045225 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:22.045197 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb"] Oct 31 16:29:22.045446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:22.045425 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" containerID="cri-o://7f8002e6c17fe3dd4993f9423b6da933e0dd99bc6365632b3bacbd4406395491" gracePeriod=30 Oct 31 16:29:22.188322 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:22.188280 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:22.497038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:22.496997 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:22.673892 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:22.673842 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:22.843645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:22.843561 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:23.616167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:23.616099 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:25.725633 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:25.725596 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:26.396580 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:26.396535 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:26.598382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:26.598345 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:26.768970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:26.768872 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:27.212927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.212892 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:27.310019 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.309976 4880 generic.go:356] "Generic (PLEG): container finished" podID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerID="7f8002e6c17fe3dd4993f9423b6da933e0dd99bc6365632b3bacbd4406395491" exitCode=0 Oct 31 16:29:27.310205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.310026 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" event={"ID":"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5","Type":"ContainerDied","Data":"7f8002e6c17fe3dd4993f9423b6da933e0dd99bc6365632b3bacbd4406395491"} Oct 31 16:29:27.353069 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.353043 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:29:27.401294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401259 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-kube-api-access-xwxmx\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-kube-api-access-xwxmx\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401347 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istiod-ca-cert\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401447 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-token\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-token\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401498 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-certs\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401555 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-credential-socket\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401612 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-envoy\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401681 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-podinfo\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401736 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-socket\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.401910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401805 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-certs" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:27.401910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401840 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istiod-ca-cert" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:29:27.401910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.401884 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-data\") pod \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\" (UID: \"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5\") " Oct 31 16:29:27.402506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.402474 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-credential-socket" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:27.402749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.402703 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-data" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:27.402975 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.402951 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.403046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.402983 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.403046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.403000 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.403046 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.403015 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.404326 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.404282 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-socket" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:27.404433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.404400 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-token" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:29:27.404530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.404502 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-podinfo" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:29:27.404916 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.404890 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-envoy" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:29:27.405036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.404999 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-kube-api-access-xwxmx" pod "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" (UID: "2e2a16d2-824c-460f-bf6d-0fba9cbb47b5"). InnerVolumeSpecName "kube-api-access-xwxmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:29:27.504882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.504847 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxmx\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-kube-api-access-xwxmx\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.504882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.504881 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.504882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.504892 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.505233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.504901 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:27.505233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:27.504910 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:29:28.319281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:28.319254 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" Oct 31 16:29:28.319671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:28.319288 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb" event={"ID":"2e2a16d2-824c-460f-bf6d-0fba9cbb47b5","Type":"ContainerDied","Data":"6b53c2e120a42449a292649f10531c65cc9a8514b69a32667f89942850866ea7"} Oct 31 16:29:28.319671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:28.319346 4880 scope.go:122] "RemoveContainer" containerID="7f8002e6c17fe3dd4993f9423b6da933e0dd99bc6365632b3bacbd4406395491" Oct 31 16:29:28.521182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:28.521123 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb"] Oct 31 16:29:28.566397 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:28.566368 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-wxsbb"] Oct 31 16:29:28.869985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:28.869931 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:29.068584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:29.068556 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:29.092367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:29.092334 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" path="/var/lib/kubelet/pods/2e2a16d2-824c-460f-bf6d-0fba9cbb47b5/volumes" Oct 31 16:29:29.250171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:29.233566 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:31.373191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:31.373158 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:31.532990 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:31.532951 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:32.128150 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:32.128097 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:32.930745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:32.930710 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:33.090937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:33.090878 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:33.882013 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:29:33.881926 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:29:34.128749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:34.128706 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:39.384647 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:39.384611 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:39.547704 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:39.547670 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:39.712415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:39.712326 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:44.440952 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:44.440918 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:46.276577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:46.276540 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:46.478534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:46.478501 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:49.479915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:49.479879 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:51.341762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:51.341712 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:51.824970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:51.824936 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:52.421329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:52.421188 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:53.179428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:53.179394 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:54.785440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:54.785407 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:56.640725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:56.640688 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:56.834740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:56.834692 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:29:56.995688 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:29:56.995607 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:00.151851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:00.151813 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:00.467001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:00.466927 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:00.619973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:00.619924 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:03.885507 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:30:03.885458 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:30:05.440683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:05.440643 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:10.767750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:10.767714 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:15.650853 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:15.650778 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:20.334753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:20.334696 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:20.519105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:20.519068 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:24.742733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:24.742701 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm"] Oct 31 16:30:24.743111 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:24.742902 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" podUID="2f1fa89b-f6d9-4a37-bd00-f86759c8b649" containerName="maas-api" containerID="cri-o://81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b" gracePeriod=30 Oct 31 16:30:24.865357 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:24.865321 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:25.133522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.133492 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:30:25.166159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.162625 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-metrics-certs\") pod \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " Oct 31 16:30:25.166159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.162733 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls\") pod \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " Oct 31 16:30:25.166159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.162820 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-openshift-service-ca\") pod \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " Oct 31 16:30:25.166159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.163103 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-kube-api-access-2h9qr\" (UniqueName: \"kubernetes.io/projected/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-kube-api-access-2h9qr\") pod \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\" (UID: \"2f1fa89b-f6d9-4a37-bd00-f86759c8b649\") " Oct 31 16:30:25.183273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.182426 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-openshift-service-ca" pod "2f1fa89b-f6d9-4a37-bd00-f86759c8b649" (UID: "2f1fa89b-f6d9-4a37-bd00-f86759c8b649"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:30:25.190474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.190426 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls" pod "2f1fa89b-f6d9-4a37-bd00-f86759c8b649" (UID: "2f1fa89b-f6d9-4a37-bd00-f86759c8b649"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:30:25.191112 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.191074 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-kube-api-access-2h9qr" pod "2f1fa89b-f6d9-4a37-bd00-f86759c8b649" (UID: "2f1fa89b-f6d9-4a37-bd00-f86759c8b649"). InnerVolumeSpecName "kube-api-access-2h9qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:30:25.200543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.199292 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-metrics-certs" pod "2f1fa89b-f6d9-4a37-bd00-f86759c8b649" (UID: "2f1fa89b-f6d9-4a37-bd00-f86759c8b649"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:30:25.266491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.266456 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9qr\" (UniqueName: \"kubernetes.io/projected/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-kube-api-access-2h9qr\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:30:25.266713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.266697 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:30:25.266810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.266798 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:30:25.266896 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.266884 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1fa89b-f6d9-4a37-bd00-f86759c8b649-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:30:25.875668 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.875632 4880 generic.go:356] "Generic (PLEG): container finished" podID="2f1fa89b-f6d9-4a37-bd00-f86759c8b649" containerID="81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b" exitCode=0 Oct 31 16:30:25.876045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.875750 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" event={"ID":"2f1fa89b-f6d9-4a37-bd00-f86759c8b649","Type":"ContainerDied","Data":"81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b"} Oct 31 16:30:25.876045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.875815 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" event={"ID":"2f1fa89b-f6d9-4a37-bd00-f86759c8b649","Type":"ContainerDied","Data":"d9ccdd07dab58402b054eb50d7ef2a366595019318b9126fbece59bf8d3728c1"} Oct 31 16:30:25.876045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.875837 4880 scope.go:122] "RemoveContainer" containerID="81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b" Oct 31 16:30:25.876045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.875967 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm" Oct 31 16:30:25.890637 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.890615 4880 scope.go:122] "RemoveContainer" containerID="81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b" Oct 31 16:30:25.890895 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:30:25.890873 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b\": container with ID starting with 81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b not found: ID does not exist" containerID="81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b" Oct 31 16:30:25.890937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:25.890905 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b"} err="failed to get container status \"81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b\": rpc error: code = NotFound desc = could not find container \"81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b\": container with ID starting with 81a84e26eff2f95545deb4a21a5c17017d29ebd376c9f4f80f89deb6dc5a142b not found: ID does not exist" Oct 31 16:30:26.032854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:26.032828 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm"] Oct 31 16:30:26.078915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:26.078887 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a7907d70-7c4f868f7f-czsfm"] Oct 31 16:30:26.406946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:26.406914 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:26.568081 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:26.568013 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:26.712899 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:26.712812 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:27.091515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:27.091439 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1fa89b-f6d9-4a37-bd00-f86759c8b649" path="/var/lib/kubelet/pods/2f1fa89b-f6d9-4a37-bd00-f86759c8b649/volumes" Oct 31 16:30:27.326817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:27.326780 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:31.782533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:31.782473 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:31.978279 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:31.977959 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:32.184618 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:32.184583 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:32.472582 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:32.472502 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:33.146084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:33.146054 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:33.343316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:33.343284 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:33.505119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:33.505087 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:33.901916 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:30:33.901873 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:30:34.729893 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:34.729856 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:35.584624 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:35.584583 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:36.108328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:36.108286 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:37.125959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:37.125926 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:37.318373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:37.318311 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:40.065959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:40.065922 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:40.333937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:40.333841 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:40.546727 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:40.546691 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:42.566863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:42.566824 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:45.366292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:45.366255 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:45.582585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:45.582510 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:47.533750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:47.533720 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:52.344731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:52.344700 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:30:57.167619 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:30:57.167586 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:02.145401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:02.145368 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:03.282579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:03.282546 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:03.881204 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:31:03.881170 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:31:06.747959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:06.747920 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:07.256402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:07.255191 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:11.993790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:11.993750 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:12.151817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:12.151778 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:12.294408 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:12.294320 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:16.944907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:16.944876 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:21.799803 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:21.799768 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:26.764033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:26.763981 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:31.555667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:31.555618 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:32.784648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:32.784607 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:33.845304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.845270 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff"] Oct 31 16:31:33.846045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846025 4880 cpu_manager.go:414] "containerMap: removing container" podUID="2f1fa89b-f6d9-4a37-bd00-f86759c8b649" containerName="maas-api" Oct 31 16:31:33.846128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846046 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1fa89b-f6d9-4a37-bd00-f86759c8b649" containerName="maas-api" Oct 31 16:31:33.846128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846061 4880 cpu_manager.go:414] "containerMap: removing container" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" Oct 31 16:31:33.846128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846066 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" Oct 31 16:31:33.846128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846075 4880 cpu_manager.go:414] "containerMap: removing container" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" Oct 31 16:31:33.846128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846080 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" Oct 31 16:31:33.846398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846233 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" Oct 31 16:31:33.846398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846245 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1fa89b-f6d9-4a37-bd00-f86759c8b649" containerName="maas-api" Oct 31 16:31:33.846398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846250 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2a16d2-824c-460f-bf6d-0fba9cbb47b5" containerName="istio-proxy" Oct 31 16:31:33.846398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.846256 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" Oct 31 16:31:33.850122 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.850105 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.881954 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:31:33.881921 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:31:33.951382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.951347 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff"] Oct 31 16:31:33.972960 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.972924 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973003 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973206 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973277 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973259 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jx2x\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-kube-api-access-9jx2x\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973396 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973447 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973519 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973599 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.973793 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.973772 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:33.990665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:33.990640 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-f43174-openshift-default-dockercfg-9wjgc\"" Oct 31 16:31:34.075224 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075194 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.075393 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075319 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.075393 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075367 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.075524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075451 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.075524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075490 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jx2x\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-kube-api-access-9jx2x\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.075646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075554 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.075646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075611 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.075646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075625 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.076029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.075943 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.076029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.076025 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.076267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.076064 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.076391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.076374 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.076478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.076459 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.076596 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.076573 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.077773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.077755 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.078029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.078009 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.133678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.133620 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jx2x\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-kube-api-access-9jx2x\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.138114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.138080 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.163972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.163944 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:34.600885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.600853 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff"] Oct 31 16:31:34.604512 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:31:34.604483 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba66155_1107_4b7a_9a97_5f0ef951e1cb.slice/crio-3643468e66a0fc5c4290b738827bed52b647a753b31ef08ef322d66e21831a91 WatchSource:0}: Error finding container 3643468e66a0fc5c4290b738827bed52b647a753b31ef08ef322d66e21831a91: Status 404 returned error can't find the container with id 3643468e66a0fc5c4290b738827bed52b647a753b31ef08ef322d66e21831a91 Oct 31 16:31:34.608071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.608030 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:31:34.608195 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.608123 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:31:34.608247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.608230 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:31:34.723909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.723871 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" event={"ID":"4ba66155-1107-4b7a-9a97-5f0ef951e1cb","Type":"ContainerStarted","Data":"955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5"} Oct 31 16:31:34.724096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.723934 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" event={"ID":"4ba66155-1107-4b7a-9a97-5f0ef951e1cb","Type":"ContainerStarted","Data":"3643468e66a0fc5c4290b738827bed52b647a753b31ef08ef322d66e21831a91"} Oct 31 16:31:34.839125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:34.839069 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podStartSLOduration=1.839053833 podStartE2EDuration="1.839053833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:34.833327228 +0000 UTC m=+1741.868622033" watchObservedRunningTime="2026-10-31 16:31:34.839053833 +0000 UTC m=+1741.874348637" Oct 31 16:31:35.164730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:35.164703 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:35.166079 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:35.166053 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 16:31:35.166215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:35.166100 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 16:31:36.164978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:36.164945 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 16:31:36.165391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:36.165002 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 16:31:36.394485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:36.394449 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:37.164539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:37.164507 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 16:31:37.164722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:37.164583 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 16:31:38.164659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:38.164628 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 16:31:38.165045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:38.164688 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 16:31:39.165238 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:39.165208 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 16:31:39.165612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:39.165269 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 16:31:40.045893 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:40.045856 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:40.164818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:40.164786 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 16:31:40.164976 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:40.164854 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 16:31:41.169547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:41.169515 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:41.170017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:41.169946 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:41.170652 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:41.170632 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:31:41.584922 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:41.584799 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:46.099930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:46.099845 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:49.101401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:49.101365 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:49.548455 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:49.548419 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:50.189181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:50.187778 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:50.433157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:50.429751 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:51.530460 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:51.530415 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:53.027947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:53.027910 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j"] Oct 31 16:31:53.028369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:53.028163 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" containerID="cri-o://3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055" gracePeriod=30 Oct 31 16:31:53.130801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:53.130770 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps"] Oct 31 16:31:53.130988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:53.130968 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" containerID="cri-o://8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c" gracePeriod=30 Oct 31 16:31:54.351192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:54.351114 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:54.747844 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:54.747810 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:54.904294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:54.904261 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:55.075331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:55.075240 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:55.595518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:55.595475 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:55.885566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:55.885524 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:57.009844 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:57.009804 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:31:57.128071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:57.128033 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:31:57.128448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:57.128408 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:31:58.307742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.307676 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:31:58.375118 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.375082 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-data\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.375315 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.375196 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-podinfo\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.375482 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.375453 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-data" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.375878 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.375846 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-token\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-token\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.375959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.375893 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-certs\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.376228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.376187 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-certs" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.376467 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.376451 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-credential-socket\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.376937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.376919 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-envoy\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.377384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.377366 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-kube-api-access-qvdkr\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-kube-api-access-qvdkr\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.377599 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.376808 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-credential-socket" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.378114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.377841 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-podinfo" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:31:58.378114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.378088 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-socket" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.379416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.379396 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-socket\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.379542 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.379528 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istiod-ca-cert\") pod \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\" (UID: \"8bc87830-f90e-40d8-a712-b0ecd2cbcd94\") " Oct 31 16:31:58.380610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.380590 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.380762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.380750 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.380894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.380881 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.381085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.381072 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.381317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.381300 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.381437 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.379889 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istiod-ca-cert" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:31:58.383696 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.383676 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-kube-api-access-qvdkr" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "kube-api-access-qvdkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:31:58.384545 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.384509 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-envoy" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.384642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.384550 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-token" pod "8bc87830-f90e-40d8-a712-b0ecd2cbcd94" (UID: "8bc87830-f90e-40d8-a712-b0ecd2cbcd94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:31:58.409119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.409098 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:31:58.482110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482077 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-credential-socket\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482121 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-data\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482165 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istiod-ca-cert\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482268 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-token\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-token\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482297 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-kube-api-access-dnhqz\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-kube-api-access-dnhqz\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482345 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-podinfo\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482392 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-data" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.482548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482401 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-credential-socket" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.482548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482418 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-envoy\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482484 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-socket\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482803 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482666 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-certs\") pod \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\" (UID: \"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d\") " Oct 31 16:31:58.482803 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482758 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istiod-ca-cert" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:31:58.482937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.482901 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-socket" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.483490 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483457 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.483490 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483486 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.483650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483501 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.483650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483516 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.483650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483530 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.483650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483545 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.483650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483559 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvdkr\" (UniqueName: \"kubernetes.io/projected/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-kube-api-access-qvdkr\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.483650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.483573 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc87830-f90e-40d8-a712-b0ecd2cbcd94-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.484655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.484619 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-certs" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.484954 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.484933 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-kube-api-access-dnhqz" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "kube-api-access-dnhqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:31:58.485047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.484967 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-envoy" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:31:58.485095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.485054 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-podinfo" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:31:58.485259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.485232 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-token" pod "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" (UID: "0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:31:58.584653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.584576 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.584653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.584606 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhqz\" (UniqueName: \"kubernetes.io/projected/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-kube-api-access-dnhqz\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.584653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.584617 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.584653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.584626 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.584653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.584635 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:31:58.933800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.933761 4880 generic.go:356] "Generic (PLEG): container finished" podID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerID="3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055" exitCode=0 Oct 31 16:31:58.933955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.933827 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" Oct 31 16:31:58.933955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.933844 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" event={"ID":"8bc87830-f90e-40d8-a712-b0ecd2cbcd94","Type":"ContainerDied","Data":"3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055"} Oct 31 16:31:58.933955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.933926 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j" event={"ID":"8bc87830-f90e-40d8-a712-b0ecd2cbcd94","Type":"ContainerDied","Data":"97d03e6115446371472931ee17678211b924eab33c7c2f7e265eeb63fd6cb492"} Oct 31 16:31:58.934186 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.933960 4880 scope.go:122] "RemoveContainer" containerID="3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055" Oct 31 16:31:58.935472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.935443 4880 generic.go:356] "Generic (PLEG): container finished" podID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerID="8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c" exitCode=0 Oct 31 16:31:58.935636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.935534 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" Oct 31 16:31:58.935636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.935543 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" event={"ID":"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d","Type":"ContainerDied","Data":"8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c"} Oct 31 16:31:58.935636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.935587 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps" event={"ID":"0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d","Type":"ContainerDied","Data":"79398cbe4181b5259d89a6c1e2f3f434ff7e357c7e6d609c27e0b9a9b43ddba3"} Oct 31 16:31:58.947815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.947798 4880 scope.go:122] "RemoveContainer" containerID="3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055" Oct 31 16:31:58.948073 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:31:58.948055 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055\": container with ID starting with 3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055 not found: ID does not exist" containerID="3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055" Oct 31 16:31:58.948121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.948083 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055"} err="failed to get container status \"3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055\": rpc error: code = NotFound desc = could not find container \"3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055\": container with ID starting with 3a3c12e43b87a48d0ef1b3c193c8b1b1029b1d6332ea1f60cbb2bb29e3af3055 not found: ID does not exist" Oct 31 16:31:58.948121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.948098 4880 scope.go:122] "RemoveContainer" containerID="8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c" Oct 31 16:31:58.957566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.957550 4880 scope.go:122] "RemoveContainer" containerID="8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c" Oct 31 16:31:58.957829 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:31:58.957809 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c\": container with ID starting with 8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c not found: ID does not exist" containerID="8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c" Oct 31 16:31:58.957883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:58.957835 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c"} err="failed to get container status \"8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c\": rpc error: code = NotFound desc = could not find container \"8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c\": container with ID starting with 8123d6c9a3d5b837b506686576b0d4534d2a4afaed29ceff6fc55033ecf3664c not found: ID does not exist" Oct 31 16:31:59.094166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:59.094109 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps"] Oct 31 16:31:59.134107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:59.134079 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-d4cps"] Oct 31 16:31:59.235818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:59.235587 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j"] Oct 31 16:31:59.239879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:59.239854 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-8569j"] Oct 31 16:31:59.799481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:31:59.799451 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:00.629059 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:00.629021 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:00.786972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:00.786920 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:00.948236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:00.948195 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:01.182164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:01.174451 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" path="/var/lib/kubelet/pods/0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d/volumes" Oct 31 16:32:01.182164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:01.175118 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" path="/var/lib/kubelet/pods/8bc87830-f90e-40d8-a712-b0ecd2cbcd94/volumes" Oct 31 16:32:01.300859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:01.294097 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:01.665197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:01.663653 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:01.850803 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:01.850767 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:02.097213 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:02.094621 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:02.795496 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:02.795462 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:02.962237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:02.962202 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:03.111613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:03.111508 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:03.880950 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:32:03.880915 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:32:05.948682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:05.948650 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:07.102919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:07.102880 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:07.272419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:07.272384 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:07.423422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:07.423384 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:08.154784 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:08.154754 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:10.855693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:10.855656 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:11.028876 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:11.028842 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:11.245280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:11.245243 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:13.186181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.186106 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:13.508089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.508007 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:13.663321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.663276 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:13.665710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.665683 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb"] Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666706 4880 cpu_manager.go:414] "containerMap: removing container" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666726 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666751 4880 cpu_manager.go:414] "containerMap: removing container" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666759 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666790 4880 cpu_manager.go:414] "containerMap: removing container" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666797 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfdc58e-fcd7-4f39-9723-ec160c7bf37f" containerName="istio-proxy" Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666944 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc87830-f90e-40d8-a712-b0ecd2cbcd94" containerName="istio-proxy" Oct 31 16:32:13.667267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.666956 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="0588aa5a-9e9d-4d41-950a-ad8a4d8fed1d" containerName="istio-proxy" Oct 31 16:32:13.672844 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.672816 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:13.799215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.795979 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3dc1da-dockercfg-qrkv9\"" Oct 31 16:32:13.830648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.810822 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3dc1da\"" Oct 31 16:32:13.851223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.842661 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb"] Oct 31 16:32:13.889888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.889845 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b9172-470f-4212-b53c-3887455786fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:13.890215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.890192 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgd76\" (UniqueName: \"kubernetes.io/projected/739b9172-470f-4212-b53c-3887455786fc-kube-api-access-sgd76\") pod \"payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:14.011297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.996206 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b9172-470f-4212-b53c-3887455786fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:14.011297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.996324 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgd76\" (UniqueName: \"kubernetes.io/projected/739b9172-470f-4212-b53c-3887455786fc-kube-api-access-sgd76\") pod \"payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:14.011297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:13.998045 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b9172-470f-4212-b53c-3887455786fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:14.097354 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:14.097277 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgd76\" (UniqueName: \"kubernetes.io/projected/739b9172-470f-4212-b53c-3887455786fc-kube-api-access-sgd76\") pod \"payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:14.297766 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:14.297730 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:14.709578 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:14.709541 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb"] Oct 31 16:32:14.714353 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:32:14.714321 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739b9172_470f_4212_b53c_3887455786fc.slice/crio-7a163150d212234c83d4eac1b8c84c6ceb6b6e3d2988e90372e365dea5933170 WatchSource:0}: Error finding container 7a163150d212234c83d4eac1b8c84c6ceb6b6e3d2988e90372e365dea5933170: Status 404 returned error can't find the container with id 7a163150d212234c83d4eac1b8c84c6ceb6b6e3d2988e90372e365dea5933170 Oct 31 16:32:15.137638 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:15.137607 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" event={"ID":"739b9172-470f-4212-b53c-3887455786fc","Type":"ContainerStarted","Data":"e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9"} Oct 31 16:32:15.137837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:15.137667 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" event={"ID":"739b9172-470f-4212-b53c-3887455786fc","Type":"ContainerStarted","Data":"7a163150d212234c83d4eac1b8c84c6ceb6b6e3d2988e90372e365dea5933170"} Oct 31 16:32:15.137837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:15.137695 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:15.244941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:15.244883 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" podStartSLOduration=2.244869955 podStartE2EDuration="2.244869955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:15.241972373 +0000 UTC m=+1782.277267178" watchObservedRunningTime="2026-10-31 16:32:15.244869955 +0000 UTC m=+1782.280164777" Oct 31 16:32:17.106971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:17.106931 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:17.302263 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:17.302230 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:17.455901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:17.455862 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:19.151926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.151892 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:19.199548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.199516 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w"] Oct 31 16:32:19.203733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.203711 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.302778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.302746 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303022 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.302992 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.303099 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.303159 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97g5\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-kube-api-access-z97g5\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.303240 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.303301 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.303355 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.303381 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.303513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.303432 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d7718-5378-4a24-baab-96fbaf7f8e80-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.400378 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.400341 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w"] Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405341 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405413 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97g5\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-kube-api-access-z97g5\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405515 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405589 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405660 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405699 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405764 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d7718-5378-4a24-baab-96fbaf7f8e80-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405882 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.405905 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.406465 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-credential-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.407067 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-socket\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.407338 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-data\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.407877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.407392 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-certs\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.408552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.407981 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d7718-5378-4a24-baab-96fbaf7f8e80-istiod-ca-cert\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.409737 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.409713 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-envoy\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.409901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.409881 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-podinfo\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.460267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.460232 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-token\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.460739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.460711 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97g5\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-kube-api-access-z97g5\") pod \"e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.519056 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.519025 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:19.950594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.950558 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w"] Oct 31 16:32:19.954245 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:32:19.954210 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07d7718_5378_4a24_baab_96fbaf7f8e80.slice/crio-555a5d65ec89cbd6670181ba1e5f7f3692edc918b90af0f18df796f6b53cb300 WatchSource:0}: Error finding container 555a5d65ec89cbd6670181ba1e5f7f3692edc918b90af0f18df796f6b53cb300: Status 404 returned error can't find the container with id 555a5d65ec89cbd6670181ba1e5f7f3692edc918b90af0f18df796f6b53cb300 Oct 31 16:32:19.956959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.956926 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:32:19.957059 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.956998 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:32:19.957059 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:19.957026 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:32:20.180001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:20.179962 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" event={"ID":"f07d7718-5378-4a24-baab-96fbaf7f8e80","Type":"ContainerStarted","Data":"cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd"} Oct 31 16:32:20.180510 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:20.180037 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" event={"ID":"f07d7718-5378-4a24-baab-96fbaf7f8e80","Type":"ContainerStarted","Data":"555a5d65ec89cbd6670181ba1e5f7f3692edc918b90af0f18df796f6b53cb300"} Oct 31 16:32:20.295981 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:20.295866 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podStartSLOduration=1.295849939 podStartE2EDuration="1.295849939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:20.293806286 +0000 UTC m=+1787.329101093" watchObservedRunningTime="2026-10-31 16:32:20.295849939 +0000 UTC m=+1787.331144743" Oct 31 16:32:20.519558 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:20.519525 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:21.537157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:21.521496 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 16:32:21.537157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:21.521568 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 16:32:22.191446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.191407 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:22.420165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.418054 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:22.519632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.519551 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 16:32:22.519788 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.519627 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 16:32:22.799560 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.799090 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv"] Oct 31 16:32:22.805189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.805163 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:22.816655 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.816622 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv"] Oct 31 16:32:22.872162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.860593 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bdc828\"" Oct 31 16:32:22.872162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.861535 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bdc828-dockercfg-xc7gm\"" Oct 31 16:32:22.998344 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.998249 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdxl\" (UniqueName: \"kubernetes.io/projected/18fd4ac2-abea-482c-a786-4e75e5d06b64-kube-api-access-wxdxl\") pod \"payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:22.998542 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:22.998433 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd4ac2-abea-482c-a786-4e75e5d06b64-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:23.100032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.099940 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd4ac2-abea-482c-a786-4e75e5d06b64-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:23.101319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.101294 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdxl\" (UniqueName: \"kubernetes.io/projected/18fd4ac2-abea-482c-a786-4e75e5d06b64-kube-api-access-wxdxl\") pod \"payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:23.101633 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.101024 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd4ac2-abea-482c-a786-4e75e5d06b64-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:23.200959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.200882 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdxl\" (UniqueName: \"kubernetes.io/projected/18fd4ac2-abea-482c-a786-4e75e5d06b64-kube-api-access-wxdxl\") pod \"payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:23.300681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.300649 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:23.519842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.519811 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 16:32:23.520051 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.519890 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 16:32:23.709665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:23.709589 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv"] Oct 31 16:32:23.713923 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:32:23.713893 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fd4ac2_abea_482c_a786_4e75e5d06b64.slice/crio-265b5a40eb43f524ff1fa71eb8c636f64bad0a6a757a92fe27d2532421b72e4a WatchSource:0}: Error finding container 265b5a40eb43f524ff1fa71eb8c636f64bad0a6a757a92fe27d2532421b72e4a: Status 404 returned error can't find the container with id 265b5a40eb43f524ff1fa71eb8c636f64bad0a6a757a92fe27d2532421b72e4a Oct 31 16:32:24.231160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:24.221229 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" event={"ID":"18fd4ac2-abea-482c-a786-4e75e5d06b64","Type":"ContainerStarted","Data":"2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb"} Oct 31 16:32:24.231160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:24.221294 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:24.231160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:24.221319 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" event={"ID":"18fd4ac2-abea-482c-a786-4e75e5d06b64","Type":"ContainerStarted","Data":"265b5a40eb43f524ff1fa71eb8c636f64bad0a6a757a92fe27d2532421b72e4a"} Oct 31 16:32:24.348586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:24.348531 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" podStartSLOduration=2.348517304 podStartE2EDuration="2.348517304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:24.332545442 +0000 UTC m=+1791.367840258" watchObservedRunningTime="2026-10-31 16:32:24.348517304 +0000 UTC m=+1791.383812108" Oct 31 16:32:24.480671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:24.480626 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:24.519405 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:24.519326 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 16:32:24.519405 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:24.519374 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 16:32:25.540216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:25.520225 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 31 16:32:25.540216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:25.520305 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 31 16:32:26.149586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:26.149558 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:32:26.524593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:26.524518 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:26.524955 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:26.524933 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:26.525733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:26.525717 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:32:29.290670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:29.290624 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:33.096570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:33.096534 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:33.378153 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:33.378061 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:32:33.382626 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:33.382603 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:32:33.385615 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:33.385597 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:32:33.390338 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:33.390318 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:32:33.397764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:33.397741 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:32:33.402163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:33.402129 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:32:33.880484 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:32:33.880447 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:32:34.089868 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:34.089835 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:35.262233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:35.262207 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:32:38.882591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:38.882557 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:43.867498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:43.867461 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:45.548038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:45.548009 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:46.954068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:46.954035 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:47.577448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:47.577416 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:48.127634 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:48.127593 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:49.029591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:49.029509 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:52.613174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:52.602990 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:52.807215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:52.807177 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:52.983932 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:52.983892 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:54.450582 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:54.450542 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:55.765745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:55.765711 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:55.924277 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:55.916146 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:32:56.137446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.137404 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr"] Oct 31 16:32:56.142661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.142636 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.177885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.177847 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.179969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.179941 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbx2\" (UniqueName: \"kubernetes.io/projected/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-kube-api-access-8qbx2\") pod \"payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.238160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.220374 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr"] Oct 31 16:32:56.238160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.237153 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a2ae37\"" Oct 31 16:32:56.238160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.237333 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a2ae37-dockercfg-nxk2n\"" Oct 31 16:32:56.313093 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.313056 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.329202 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.318125 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.329446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.327582 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l"] Oct 31 16:32:56.349628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.349562 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.351062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.351020 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbx2\" (UniqueName: \"kubernetes.io/projected/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-kube-api-access-8qbx2\") pod \"payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.367907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.367851 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l"] Oct 31 16:32:56.438953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.438896 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbx2\" (UniqueName: \"kubernetes.io/projected/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-kube-api-access-8qbx2\") pod \"payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.454042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.453850 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x896\" (UniqueName: \"kubernetes.io/projected/888e0fd6-1a89-41e5-a51b-4c506c1f9399-kube-api-access-9x896\") pod \"payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.454282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.454073 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888e0fd6-1a89-41e5-a51b-4c506c1f9399-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.490296 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.490004 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:56.566158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.564205 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x896\" (UniqueName: \"kubernetes.io/projected/888e0fd6-1a89-41e5-a51b-4c506c1f9399-kube-api-access-9x896\") pod \"payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.566782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.566755 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888e0fd6-1a89-41e5-a51b-4c506c1f9399-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.567905 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.567880 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888e0fd6-1a89-41e5-a51b-4c506c1f9399-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.619505 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.619440 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x896\" (UniqueName: \"kubernetes.io/projected/888e0fd6-1a89-41e5-a51b-4c506c1f9399-kube-api-access-9x896\") pod \"payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.729406 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.729375 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:56.999335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:56.999307 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr"] Oct 31 16:32:57.112834 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.112803 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l"] Oct 31 16:32:57.118132 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:32:57.118104 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888e0fd6_1a89_41e5_a51b_4c506c1f9399.slice/crio-17bdc03a1464b13108b18937e50da0caf6031c5bb5ef1eaacf13ccd7e58cfe73 WatchSource:0}: Error finding container 17bdc03a1464b13108b18937e50da0caf6031c5bb5ef1eaacf13ccd7e58cfe73: Status 404 returned error can't find the container with id 17bdc03a1464b13108b18937e50da0caf6031c5bb5ef1eaacf13ccd7e58cfe73 Oct 31 16:32:57.562903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.562871 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" event={"ID":"8c1cf374-95c4-47e7-9ed3-5b38110a9c60","Type":"ContainerStarted","Data":"c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883"} Oct 31 16:32:57.563076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.562935 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:32:57.563076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.562961 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" event={"ID":"8c1cf374-95c4-47e7-9ed3-5b38110a9c60","Type":"ContainerStarted","Data":"d9f1ea0498c36412753edddd5ce70d5cff0cc97f2b3271009119f17450a4483c"} Oct 31 16:32:57.564909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.564880 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" event={"ID":"888e0fd6-1a89-41e5-a51b-4c506c1f9399","Type":"ContainerStarted","Data":"f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8"} Oct 31 16:32:57.565022 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.564929 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" event={"ID":"888e0fd6-1a89-41e5-a51b-4c506c1f9399","Type":"ContainerStarted","Data":"17bdc03a1464b13108b18937e50da0caf6031c5bb5ef1eaacf13ccd7e58cfe73"} Oct 31 16:32:57.565022 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.564951 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:32:57.668190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.668132 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" podStartSLOduration=1.668118739 podStartE2EDuration="1.668118739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:57.667449951 +0000 UTC m=+1824.702744754" watchObservedRunningTime="2026-10-31 16:32:57.668118739 +0000 UTC m=+1824.703413542" Oct 31 16:32:57.778494 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:57.778099 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" podStartSLOduration=1.778082258 podStartE2EDuration="1.778082258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:57.771685078 +0000 UTC m=+1824.806979881" watchObservedRunningTime="2026-10-31 16:32:57.778082258 +0000 UTC m=+1824.813377063" Oct 31 16:32:59.854809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:32:59.854775 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:03.881726 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:03.881691 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:33:05.066676 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:05.066647 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:08.575716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:08.575685 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:33:08.576171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:08.575757 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:33:09.844519 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:09.844487 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:14.621011 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:14.620924 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:15.443893 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:15.443862 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:19.389130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:19.389076 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:23.890266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:23.890210 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq"] Oct 31 16:33:23.894650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:23.894629 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:23.946902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:23.946870 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq"] Oct 31 16:33:23.988278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:23.988249 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-487213-serving-cert\"" Oct 31 16:33:24.032341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.032300 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.032598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.032581 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.032660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.032630 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.032812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.032782 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-kube-api-access-ccrmh\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.097188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.097153 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:24.134054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.134027 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.134301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.134072 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.134301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.134109 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-kube-api-access-ccrmh\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.134301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.134219 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.134498 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:24.134350 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-serving-cert: secret "maas-api-e2e-shared-b-w0-487213-serving-cert" not found Oct 31 16:33:24.134498 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:24.134443 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls podName:6b5295a8-4227-4d5b-a12a-dfcc39f5c214 nodeName:}" failed. No retries permitted until 2026-10-31 16:33:24.634421658 +0000 UTC m=+1851.669716443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls") pod "maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" (UID: "6b5295a8-4227-4d5b-a12a-dfcc39f5c214") : secret "maas-api-e2e-shared-b-w0-487213-serving-cert" not found Oct 31 16:33:24.134933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.134913 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.137203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.137129 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.188221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.188126 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-kube-api-access-ccrmh\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.264561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.264519 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:24.401041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.400983 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:24.644114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.644080 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.646590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.646569 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:24.808709 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:24.808681 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:25.182371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:25.182338 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq"] Oct 31 16:33:25.187340 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:33:25.187303 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5295a8_4227_4d5b_a12a_dfcc39f5c214.slice/crio-bd51cad74f3c3448ca12ddce5058d8ddf735af96680d07466fe50dda096f4317 WatchSource:0}: Error finding container bd51cad74f3c3448ca12ddce5058d8ddf735af96680d07466fe50dda096f4317: Status 404 returned error can't find the container with id bd51cad74f3c3448ca12ddce5058d8ddf735af96680d07466fe50dda096f4317 Oct 31 16:33:25.188829 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:25.188805 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:33:25.801043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:25.801001 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" event={"ID":"6b5295a8-4227-4d5b-a12a-dfcc39f5c214","Type":"ContainerStarted","Data":"bd51cad74f3c3448ca12ddce5058d8ddf735af96680d07466fe50dda096f4317"} Oct 31 16:33:26.810780 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:26.810733 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" event={"ID":"6b5295a8-4227-4d5b-a12a-dfcc39f5c214","Type":"ContainerStarted","Data":"5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe"} Oct 31 16:33:26.811302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:26.810800 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:26.917389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:26.917329 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" podStartSLOduration=3.013014301 podStartE2EDuration="3.917314801s" totalImagesPullingTime="904.3005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:26.916107837 +0000 UTC m=+1853.951402640" watchObservedRunningTime="2026-10-31 16:33:26.917314801 +0000 UTC m=+1853.952609604" Oct 31 16:33:29.127055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:29.126959 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:32.824479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:32.824450 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:33:33.426462 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.426422 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:33.552401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.552364 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg"] Oct 31 16:33:33.558551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.558525 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.594912 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.594864 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zxs\" (UniqueName: \"kubernetes.io/projected/569fe801-9503-4103-b593-f064e15cf2a6-kube-api-access-p6zxs\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.595107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.595089 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.595232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.595187 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.595300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.595282 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569fe801-9503-4103-b593-f064e15cf2a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.613149 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.613112 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg"] Oct 31 16:33:33.651648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.651620 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-d0755f-serving-cert\"" Oct 31 16:33:33.696648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.696555 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569fe801-9503-4103-b593-f064e15cf2a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.696793 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.696699 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zxs\" (UniqueName: \"kubernetes.io/projected/569fe801-9503-4103-b593-f064e15cf2a6-kube-api-access-p6zxs\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.696845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.696833 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.696916 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.696899 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.697111 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:33.697084 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-serving-cert: secret "maas-api-e2e-shared-b-w4-d0755f-serving-cert" not found Oct 31 16:33:33.697265 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:33.697249 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls podName:569fe801-9503-4103-b593-f064e15cf2a6 nodeName:}" failed. No retries permitted until 2026-10-31 16:33:34.197227034 +0000 UTC m=+1861.232521823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls") pod "maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" (UID: "569fe801-9503-4103-b593-f064e15cf2a6") : secret "maas-api-e2e-shared-b-w4-d0755f-serving-cert" not found Oct 31 16:33:33.697413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.697388 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569fe801-9503-4103-b593-f064e15cf2a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.699442 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.699421 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.749690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.749656 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zxs\" (UniqueName: \"kubernetes.io/projected/569fe801-9503-4103-b593-f064e15cf2a6-kube-api-access-p6zxs\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:33.766714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.766680 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:33.887679 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:33.887633 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:33:33.926280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.926239 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:33.977150 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.976649 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7204-b7c50d"] Oct 31 16:33:33.983212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:33.983159 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:34.028961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.028927 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7204-b7c50d"] Oct 31 16:33:34.032480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.032454 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-szvbq\"" Oct 31 16:33:34.071713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.071676 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:34.102879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.102849 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vlr\" (UniqueName: \"kubernetes.io/projected/91860186-88ee-4857-a0f0-5a6887b4846e-kube-api-access-t6vlr\") pod \"test-curl-7204-b7c50d\" (UID: \"91860186-88ee-4857-a0f0-5a6887b4846e\") " pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:34.204959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.204926 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:34.205161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.204984 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vlr\" (UniqueName: \"kubernetes.io/projected/91860186-88ee-4857-a0f0-5a6887b4846e-kube-api-access-t6vlr\") pod \"test-curl-7204-b7c50d\" (UID: \"91860186-88ee-4857-a0f0-5a6887b4846e\") " pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:34.207519 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.207489 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:34.258003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.257935 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vlr\" (UniqueName: \"kubernetes.io/projected/91860186-88ee-4857-a0f0-5a6887b4846e-kube-api-access-t6vlr\") pod \"test-curl-7204-b7c50d\" (UID: \"91860186-88ee-4857-a0f0-5a6887b4846e\") " pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:34.310405 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.310367 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:34.474701 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.474671 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:34.690753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.690717 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7204-b7c50d"] Oct 31 16:33:34.695319 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:33:34.695291 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91860186_88ee_4857_a0f0_5a6887b4846e.slice/crio-23c547b66f9098531aa5ec5f06f707aebf5ae1cde70fb11e4f3c617379d4f410 WatchSource:0}: Error finding container 23c547b66f9098531aa5ec5f06f707aebf5ae1cde70fb11e4f3c617379d4f410: Status 404 returned error can't find the container with id 23c547b66f9098531aa5ec5f06f707aebf5ae1cde70fb11e4f3c617379d4f410 Oct 31 16:33:34.860104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.860022 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg"] Oct 31 16:33:34.864178 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:33:34.864118 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569fe801_9503_4103_b593_f064e15cf2a6.slice/crio-84a3e40228c45c24da76b63fd9ab5068376c68c5cfa49ade490d25d25d9290f4 WatchSource:0}: Error finding container 84a3e40228c45c24da76b63fd9ab5068376c68c5cfa49ade490d25d25d9290f4: Status 404 returned error can't find the container with id 84a3e40228c45c24da76b63fd9ab5068376c68c5cfa49ade490d25d25d9290f4 Oct 31 16:33:34.875473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.875427 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" event={"ID":"569fe801-9503-4103-b593-f064e15cf2a6","Type":"ContainerStarted","Data":"84a3e40228c45c24da76b63fd9ab5068376c68c5cfa49ade490d25d25d9290f4"} Oct 31 16:33:34.877176 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.877126 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7204-b7c50d" event={"ID":"91860186-88ee-4857-a0f0-5a6887b4846e","Type":"ContainerStarted","Data":"41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d"} Oct 31 16:33:34.877266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.877178 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7204-b7c50d" event={"ID":"91860186-88ee-4857-a0f0-5a6887b4846e","Type":"ContainerStarted","Data":"23c547b66f9098531aa5ec5f06f707aebf5ae1cde70fb11e4f3c617379d4f410"} Oct 31 16:33:34.983168 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:34.983102 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7204-b7c50d" podStartSLOduration=1.98308909 podStartE2EDuration="1.98308909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:34.981036108 +0000 UTC m=+1862.016330908" watchObservedRunningTime="2026-10-31 16:33:34.98308909 +0000 UTC m=+1862.018383904" Oct 31 16:33:35.888801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:35.888764 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" event={"ID":"569fe801-9503-4103-b593-f064e15cf2a6","Type":"ContainerStarted","Data":"7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4"} Oct 31 16:33:36.006261 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:36.006209 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" podStartSLOduration=2.097647242 podStartE2EDuration="3.006195463s" totalImagesPullingTime="908.548221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:36.002074565 +0000 UTC m=+1863.037369369" watchObservedRunningTime="2026-10-31 16:33:36.006195463 +0000 UTC m=+1863.041490278" Oct 31 16:33:36.406787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:36.406704 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7204-b7c50d"] Oct 31 16:33:36.412212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:36.412180 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7204-b7c50d"] Oct 31 16:33:36.896339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:36.896277 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7204-b7c50d" podUID="91860186-88ee-4857-a0f0-5a6887b4846e" containerName="test-curl-7204-b7c50d" containerID="cri-o://41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d" gracePeriod=2 Oct 31 16:33:36.896607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:36.896582 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:36.943731 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:36.943692 4880 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7204-b7c50d\" is forbidden: User \"system:node:ip-10-0-142-127.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-127.us-west-2.compute.internal' and this object" podUID="91860186-88ee-4857-a0f0-5a6887b4846e" pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:39.073166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.073108 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:39.177648 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.177585 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:39.297242 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.297210 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91860186-88ee-4857-a0f0-5a6887b4846e-kube-api-access-t6vlr\" (UniqueName: \"kubernetes.io/projected/91860186-88ee-4857-a0f0-5a6887b4846e-kube-api-access-t6vlr\") pod \"91860186-88ee-4857-a0f0-5a6887b4846e\" (UID: \"91860186-88ee-4857-a0f0-5a6887b4846e\") " Oct 31 16:33:39.299417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.299389 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91860186-88ee-4857-a0f0-5a6887b4846e-kube-api-access-t6vlr" pod "91860186-88ee-4857-a0f0-5a6887b4846e" (UID: "91860186-88ee-4857-a0f0-5a6887b4846e"). InnerVolumeSpecName "kube-api-access-t6vlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:33:39.399371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.399333 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vlr\" (UniqueName: \"kubernetes.io/projected/91860186-88ee-4857-a0f0-5a6887b4846e-kube-api-access-t6vlr\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:33:39.922642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.922601 4880 generic.go:356] "Generic (PLEG): container finished" podID="91860186-88ee-4857-a0f0-5a6887b4846e" containerID="41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d" exitCode=137 Oct 31 16:33:39.922642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.922644 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:39.922883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.922648 4880 scope.go:122] "RemoveContainer" containerID="41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d" Oct 31 16:33:39.933402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.933387 4880 scope.go:122] "RemoveContainer" containerID="41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d" Oct 31 16:33:39.933678 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:39.933659 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d\": container with ID starting with 41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d not found: ID does not exist" containerID="41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d" Oct 31 16:33:39.933738 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:39.933687 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d"} err="failed to get container status \"41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d\": rpc error: code = NotFound desc = could not find container \"41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d\": container with ID starting with 41320c6f1fa9ef89b00abf73536cf8e1c86537f5c37fbe262ea8920079f8815d not found: ID does not exist" Oct 31 16:33:39.978171 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:39.978125 4880 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7204-b7c50d\" is forbidden: User \"system:node:ip-10-0-142-127.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-127.us-west-2.compute.internal' and this object" podUID="91860186-88ee-4857-a0f0-5a6887b4846e" pod="opendatahub/test-curl-7204-b7c50d" Oct 31 16:33:41.092295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:41.092260 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91860186-88ee-4857-a0f0-5a6887b4846e" path="/var/lib/kubelet/pods/91860186-88ee-4857-a0f0-5a6887b4846e/volumes" Oct 31 16:33:42.912411 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:42.912381 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:33:44.091837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.091801 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs"] Oct 31 16:33:44.092566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.092548 4880 cpu_manager.go:414] "containerMap: removing container" podUID="91860186-88ee-4857-a0f0-5a6887b4846e" containerName="test-curl-7204-b7c50d" Oct 31 16:33:44.092566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.092566 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91860186-88ee-4857-a0f0-5a6887b4846e" containerName="test-curl-7204-b7c50d" Oct 31 16:33:44.092699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.092672 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="91860186-88ee-4857-a0f0-5a6887b4846e" containerName="test-curl-7204-b7c50d" Oct 31 16:33:44.097544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.097527 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.148640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.148610 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs"] Oct 31 16:33:44.177851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.177819 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c249f9-8a9a-46f7-b502-6cf2d21a2508-openshift-service-ca\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.178020 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.177897 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.178092 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.178068 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-metrics-certs\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.178214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.178198 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqmh\" (UniqueName: \"kubernetes.io/projected/83c249f9-8a9a-46f7-b502-6cf2d21a2508-kube-api-access-6jqmh\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.189131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.189102 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3dc1da-serving-cert\"" Oct 31 16:33:44.286156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.284158 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.286156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.284259 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-metrics-certs\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.286156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.284345 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqmh\" (UniqueName: \"kubernetes.io/projected/83c249f9-8a9a-46f7-b502-6cf2d21a2508-kube-api-access-6jqmh\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.286156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.284560 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c249f9-8a9a-46f7-b502-6cf2d21a2508-openshift-service-ca\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.286156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.285448 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c249f9-8a9a-46f7-b502-6cf2d21a2508-openshift-service-ca\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.286156 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:44.285566 4880 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-serving-cert: secret "maas-api-e2e-api-b-3dc1da-serving-cert" not found Oct 31 16:33:44.286156 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:33:44.285634 4880 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls podName:83c249f9-8a9a-46f7-b502-6cf2d21a2508 nodeName:}" failed. No retries permitted until 2026-10-31 16:33:44.785614709 +0000 UTC m=+1871.820909505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls") pod "maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" (UID: "83c249f9-8a9a-46f7-b502-6cf2d21a2508") : secret "maas-api-e2e-api-b-3dc1da-serving-cert" not found Oct 31 16:33:44.369260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.353761 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:44.451447 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.441917 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqmh\" (UniqueName: \"kubernetes.io/projected/83c249f9-8a9a-46f7-b502-6cf2d21a2508-kube-api-access-6jqmh\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.451724 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.441952 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-metrics-certs\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.563041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.562438 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:44.750774 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.750739 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:44.840266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.840228 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:44.843296 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:44.843265 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls\") pod \"maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:45.011301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:45.011215 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:45.432473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:45.412874 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs"] Oct 31 16:33:45.977517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:45.977483 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" event={"ID":"83c249f9-8a9a-46f7-b502-6cf2d21a2508","Type":"ContainerStarted","Data":"619dda7d7a49f3a44e167504c60a6a4619ff86801e0a064783ea02bebf07efb0"} Oct 31 16:33:46.986685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:46.986645 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" event={"ID":"83c249f9-8a9a-46f7-b502-6cf2d21a2508","Type":"ContainerStarted","Data":"8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7"} Oct 31 16:33:46.987200 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:46.986710 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:47.099343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:47.099282 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" podStartSLOduration=2.266641276 podStartE2EDuration="3.099266051s" totalImagesPullingTime="832.624775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:47.094671428 +0000 UTC m=+1874.129966214" watchObservedRunningTime="2026-10-31 16:33:47.099266051 +0000 UTC m=+1874.134560858" Oct 31 16:33:48.810367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:48.810322 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:48.961983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:48.959664 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:49.986158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:49.985394 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:51.385751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:51.385717 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:51.536129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:51.536093 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:52.999686 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:52.999659 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:33:55.065089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:55.065053 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:59.742627 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:59.742590 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:33:59.909812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:33:59.909772 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:00.155841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:00.155805 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:03.882754 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:34:03.882716 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:34:04.848019 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:04.847972 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:09.751005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:09.750961 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:14.522632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:14.522598 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:15.871835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:15.871789 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:16.072856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:16.072799 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:19.361667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:19.361627 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:24.159085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:24.159042 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:28.941265 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:28.941232 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:29.457081 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:29.457046 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:29.603454 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:29.603421 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:33.750334 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:33.750296 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:33.889241 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:34:33.889190 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:34:33.895222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:33.895186 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:34.048815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:34.048777 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:34.330352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:34.330260 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:34.467037 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:34.466994 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:38.799680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:38.799645 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:39.293343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.293313 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq"] Oct 31 16:34:39.293542 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.293508 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" containerID="cri-o://0cc30f6e9aba11e9f51d630633f76b373e392fc2d7385c75dc9b7f71318a81a0" gracePeriod=30 Oct 31 16:34:39.319804 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.319771 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz"] Oct 31 16:34:39.320040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.320007 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" containerID="cri-o://38f528cd5b7c60c95f50988ff80d01e8f6f99daf99ba12676aa6c68893144a07" gracePeriod=30 Oct 31 16:34:39.327193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.327163 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x"] Oct 31 16:34:39.327469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.327423 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" containerID="cri-o://9ebbc37f384c3b648e30d1e9d3ba66e2afabf9b8e2f75ab127477a5c6e6ea4c0" gracePeriod=30 Oct 31 16:34:39.465285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.465255 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd"] Oct 31 16:34:39.465473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:39.465452 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" containerID="cri-o://e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653" gracePeriod=30 Oct 31 16:34:42.244152 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:42.244106 4880 patch_prober.go:29] interesting pod/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:34:42.244525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:42.244183 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:34:42.290538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:42.290498 4880 patch_prober.go:29] interesting pod/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:34:42.290703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:42.290560 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:34:42.692643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:42.692613 4880 patch_prober.go:29] interesting pod/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:34:42.692806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:42.692674 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:34:43.560756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:43.560723 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:34:43.561167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:43.560803 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:34:43.638362 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:43.638332 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:44.329562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.329533 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l"] Oct 31 16:34:44.334918 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.334897 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.383797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.383767 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bdc828-openshift-default-dockercfg-gq77f\"" Oct 31 16:34:44.437539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.437512 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l"] Oct 31 16:34:44.476549 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.476504 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.476713 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.476608 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.476813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.476749 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.476813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.476787 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.476932 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.476837 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.476994 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.476974 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt8xh\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-kube-api-access-dt8xh\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.477066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.477032 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5585ac35-f2e6-4344-9d72-92ab372d5918-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.477066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.477062 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.477235 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.477109 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.516657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.516246 4880 generic.go:356] "Generic (PLEG): container finished" podID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerID="0cc30f6e9aba11e9f51d630633f76b373e392fc2d7385c75dc9b7f71318a81a0" exitCode=0 Oct 31 16:34:44.516657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.516544 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" event={"ID":"deab96e1-8cff-43b4-be2c-6a0c3b64348d","Type":"ContainerDied","Data":"0cc30f6e9aba11e9f51d630633f76b373e392fc2d7385c75dc9b7f71318a81a0"} Oct 31 16:34:44.518747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.518708 4880 generic.go:356] "Generic (PLEG): container finished" podID="15a0d474-ca84-45b8-8711-a944b019383c" containerID="9ebbc37f384c3b648e30d1e9d3ba66e2afabf9b8e2f75ab127477a5c6e6ea4c0" exitCode=0 Oct 31 16:34:44.519089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.519002 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" event={"ID":"15a0d474-ca84-45b8-8711-a944b019383c","Type":"ContainerDied","Data":"9ebbc37f384c3b648e30d1e9d3ba66e2afabf9b8e2f75ab127477a5c6e6ea4c0"} Oct 31 16:34:44.521781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.521690 4880 generic.go:356] "Generic (PLEG): container finished" podID="4180fbb7-3252-40b1-b401-34b666278d57" containerID="38f528cd5b7c60c95f50988ff80d01e8f6f99daf99ba12676aa6c68893144a07" exitCode=0 Oct 31 16:34:44.521933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.521760 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" event={"ID":"4180fbb7-3252-40b1-b401-34b666278d57","Type":"ContainerDied","Data":"38f528cd5b7c60c95f50988ff80d01e8f6f99daf99ba12676aa6c68893144a07"} Oct 31 16:34:44.592704 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.592614 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.592718 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.592871 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.592911 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.592961 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.593105 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt8xh\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-kube-api-access-dt8xh\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.593168 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5585ac35-f2e6-4344-9d72-92ab372d5918-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.593200 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.593269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.593248 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.594485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.594381 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.595266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.594838 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5585ac35-f2e6-4344-9d72-92ab372d5918-istiod-ca-cert\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.595266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.595105 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-credential-socket\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.596210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.595488 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-data\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.605108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.596667 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-podinfo\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.622900 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.615691 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-certs\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.624635 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.624609 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-envoy\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.676561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.676528 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-token\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.677525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.677502 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt8xh\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-kube-api-access-dt8xh\") pod \"e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.704867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.702801 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:34:44.709993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.709970 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:34:44.776756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.776734 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:34:44.797007 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.796983 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-credential-socket\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797132 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797017 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-kube-api-access-z9ng4\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-kube-api-access-z9ng4\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797132 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797050 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-certs\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797132 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797085 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-socket\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797132 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797104 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-token\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-token\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797190 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-data\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797218 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-kube-api-access-pcp8g\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-kube-api-access-pcp8g\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797249 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-envoy\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797266 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-credential-socket" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.797356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797303 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4180fbb7-3252-40b1-b401-34b666278d57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4180fbb7-3252-40b1-b401-34b666278d57-istio-podinfo\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797356 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797344 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-istio-token\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-istio-token\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797401 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4180fbb7-3252-40b1-b401-34b666278d57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4180fbb7-3252-40b1-b401-34b666278d57-istiod-ca-cert\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797489 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-certs\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797523 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-envoy\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797562 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-socket\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797633 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-podinfo\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797634 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-socket" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.797961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797664 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-data\") pod \"4180fbb7-3252-40b1-b401-34b666278d57\" (UID: \"4180fbb7-3252-40b1-b401-34b666278d57\") " Oct 31 16:34:44.797961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797730 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-credential-socket\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.797961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797763 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istiod-ca-cert\") pod \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\" (UID: \"deab96e1-8cff-43b4-be2c-6a0c3b64348d\") " Oct 31 16:34:44.798125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797661 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-certs" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.798125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797791 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-data" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.798125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.797959 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-data" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.798304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798106 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-socket" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.798366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798333 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-credential-socket" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.798424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798404 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4180fbb7-3252-40b1-b401-34b666278d57-istiod-ca-cert" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:34:44.798479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798456 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istiod-ca-cert" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:34:44.798707 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798687 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-certs" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.798883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798862 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.798953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798888 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.798953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798903 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4180fbb7-3252-40b1-b401-34b666278d57-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.798953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798917 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.798953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798931 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.798953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798943 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.799239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798956 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.799239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798971 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.799239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798986 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.799239 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.798999 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.800487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.800458 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-kube-api-access-pcp8g" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "kube-api-access-pcp8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:44.800613 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.800590 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4180fbb7-3252-40b1-b401-34b666278d57-istio-podinfo" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:34:44.800821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.800789 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-envoy" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.801108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.800815 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-token" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:44.801108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.800874 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-istio-token" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:44.801108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.801019 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-envoy" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.801264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.801212 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-kube-api-access-z9ng4" pod "4180fbb7-3252-40b1-b401-34b666278d57" (UID: "4180fbb7-3252-40b1-b401-34b666278d57"). InnerVolumeSpecName "kube-api-access-z9ng4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:44.801676 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.801661 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-podinfo" pod "deab96e1-8cff-43b4-be2c-6a0c3b64348d" (UID: "deab96e1-8cff-43b4-be2c-6a0c3b64348d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:34:44.817524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.817502 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:34:44.871768 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.871699 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:44.900321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.899933 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-kube-api-access-z9pvl\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-kube-api-access-z9pvl\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.899977 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-credential-socket\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900080 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-istio-token\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-istio-token\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900321 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900212 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a0d474-ca84-45b8-8711-a944b019383c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a0d474-ca84-45b8-8711-a944b019383c-istiod-ca-cert\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900328 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-data\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900436 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-certs\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900472 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-socket\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900539 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-envoy\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.900811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900628 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15a0d474-ca84-45b8-8711-a944b019383c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a0d474-ca84-45b8-8711-a944b019383c-istio-podinfo\") pod \"15a0d474-ca84-45b8-8711-a944b019383c\" (UID: \"15a0d474-ca84-45b8-8711-a944b019383c\") " Oct 31 16:34:44.901592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900537 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-credential-socket" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.901592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.900726 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-certs" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.901592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.901039 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a0d474-ca84-45b8-8711-a944b019383c-istiod-ca-cert" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:34:44.901592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.901300 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-data" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.901592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.901541 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-socket" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904884 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904913 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcp8g\" (UniqueName: \"kubernetes.io/projected/deab96e1-8cff-43b4-be2c-6a0c3b64348d-kube-api-access-pcp8g\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904930 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4180fbb7-3252-40b1-b401-34b666278d57-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904945 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4180fbb7-3252-40b1-b401-34b666278d57-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904959 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15a0d474-ca84-45b8-8711-a944b019383c-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904971 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904982 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.904994 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.905007 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deab96e1-8cff-43b4-be2c-6a0c3b64348d-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.905019 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.905039 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.905051 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.906155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.905063 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ng4\" (UniqueName: \"kubernetes.io/projected/4180fbb7-3252-40b1-b401-34b666278d57-kube-api-access-z9ng4\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:44.914937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.914599 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-envoy" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:44.914937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.914766 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-istio-token" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:44.914937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.914814 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-kube-api-access-z9pvl" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "kube-api-access-z9pvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:44.916065 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:44.916040 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15a0d474-ca84-45b8-8711-a944b019383c-istio-podinfo" pod "15a0d474-ca84-45b8-8711-a944b019383c" (UID: "15a0d474-ca84-45b8-8711-a944b019383c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:34:45.005907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.005871 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-socket\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.005961 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-podinfo\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006003 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istiod-ca-cert\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006090 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-credential-socket\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006200 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-socket" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:45.006554 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006530 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-credential-socket" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:45.006604 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006571 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istiod-ca-cert" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:34:45.006662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006641 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-kube-api-access-sttfd\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-kube-api-access-sttfd\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006757 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006741 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-data\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006839 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006779 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-envoy\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006839 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006828 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-token\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-token\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.006953 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.006874 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-certs\") pod \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\" (UID: \"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b\") " Oct 31 16:34:45.007053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007031 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-data" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:45.007286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007259 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-certs" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:45.007732 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007707 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007742 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007757 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15a0d474-ca84-45b8-8711-a944b019383c-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007770 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007785 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15a0d474-ca84-45b8-8711-a944b019383c-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007798 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9pvl\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-kube-api-access-z9pvl\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007812 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007824 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.007850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.007839 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15a0d474-ca84-45b8-8711-a944b019383c-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.009202 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.009132 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-podinfo" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:34:45.009384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.009362 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-envoy" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:34:45.009495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.009411 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-kube-api-access-sttfd" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "kube-api-access-sttfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:45.010787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.010762 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-token" pod "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" (UID: "5a05bcb5-0ec3-42ea-8c14-0e100ea8943b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:45.108995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.108962 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sttfd\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-kube-api-access-sttfd\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.109188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.108998 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.109188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.109015 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.109188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.109034 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:34:45.311286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.311252 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l"] Oct 31 16:34:45.316778 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:34:45.316750 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5585ac35_f2e6_4344_9d72_92ab372d5918.slice/crio-84e7931af2498c7cac0801116fc9f565cb9a156266bab88c5f7b2a5d6e74b23a WatchSource:0}: Error finding container 84e7931af2498c7cac0801116fc9f565cb9a156266bab88c5f7b2a5d6e74b23a: Status 404 returned error can't find the container with id 84e7931af2498c7cac0801116fc9f565cb9a156266bab88c5f7b2a5d6e74b23a Oct 31 16:34:45.319262 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.319233 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:34:45.319344 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.319296 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:34:45.319344 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.319325 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:34:45.531485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.531453 4880 generic.go:356] "Generic (PLEG): container finished" podID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerID="e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653" exitCode=0 Oct 31 16:34:45.531667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.531500 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" Oct 31 16:34:45.531667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.531531 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" event={"ID":"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b","Type":"ContainerDied","Data":"e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653"} Oct 31 16:34:45.531667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.531586 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd" event={"ID":"5a05bcb5-0ec3-42ea-8c14-0e100ea8943b","Type":"ContainerDied","Data":"725feeccd29ece5e952d58a02d93ecf33535c320977d923c2589bf3f8f8086c0"} Oct 31 16:34:45.531667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.531610 4880 scope.go:122] "RemoveContainer" containerID="e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653" Oct 31 16:34:45.533907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.533865 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" event={"ID":"5585ac35-f2e6-4344-9d72-92ab372d5918","Type":"ContainerStarted","Data":"21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9"} Oct 31 16:34:45.534025 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.533935 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" event={"ID":"5585ac35-f2e6-4344-9d72-92ab372d5918","Type":"ContainerStarted","Data":"84e7931af2498c7cac0801116fc9f565cb9a156266bab88c5f7b2a5d6e74b23a"} Oct 31 16:34:45.535400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.535382 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" Oct 31 16:34:45.535507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.535426 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x" event={"ID":"15a0d474-ca84-45b8-8711-a944b019383c","Type":"ContainerDied","Data":"622e6b1aa2f40c3e51d1aeb25eba6bc5a3e9d8291903d4d101dfa9f3807d246b"} Oct 31 16:34:45.536940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.536924 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" Oct 31 16:34:45.537068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.536948 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz" event={"ID":"4180fbb7-3252-40b1-b401-34b666278d57","Type":"ContainerDied","Data":"b5931e8d5f8621b0986562a401b2d0b08c59c80cf470794b4a5c360a048fca15"} Oct 31 16:34:45.538475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.538455 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" event={"ID":"deab96e1-8cff-43b4-be2c-6a0c3b64348d","Type":"ContainerDied","Data":"0a46b7af1c119abce0bb21c770080571bdda498407d1ab6a4127bc8da53a6299"} Oct 31 16:34:45.538561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.538481 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq" Oct 31 16:34:45.545173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.545156 4880 scope.go:122] "RemoveContainer" containerID="e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653" Oct 31 16:34:45.545439 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:34:45.545410 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653\": container with ID starting with e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653 not found: ID does not exist" containerID="e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653" Oct 31 16:34:45.545503 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.545439 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653"} err="failed to get container status \"e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653\": rpc error: code = NotFound desc = could not find container \"e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653\": container with ID starting with e802833260e56d237f8d34f4809f95e0976d04013ed958d6db9a92a2c5093653 not found: ID does not exist" Oct 31 16:34:45.545503 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.545456 4880 scope.go:122] "RemoveContainer" containerID="9ebbc37f384c3b648e30d1e9d3ba66e2afabf9b8e2f75ab127477a5c6e6ea4c0" Oct 31 16:34:45.555506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.555490 4880 scope.go:122] "RemoveContainer" containerID="38f528cd5b7c60c95f50988ff80d01e8f6f99daf99ba12676aa6c68893144a07" Oct 31 16:34:45.565042 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.565026 4880 scope.go:122] "RemoveContainer" containerID="0cc30f6e9aba11e9f51d630633f76b373e392fc2d7385c75dc9b7f71318a81a0" Oct 31 16:34:45.688359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.688326 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd"] Oct 31 16:34:45.692922 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.692896 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-884jd"] Oct 31 16:34:45.792409 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.792378 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq"] Oct 31 16:34:45.796751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.796727 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-stllq"] Oct 31 16:34:45.872150 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.872036 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:45.895971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.895945 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x"] Oct 31 16:34:45.900770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:45.900749 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-cj29x"] Oct 31 16:34:46.002127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:46.002096 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz"] Oct 31 16:34:46.006122 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:46.006098 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-pqrsz"] Oct 31 16:34:46.111579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:46.111505 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podStartSLOduration=2.111482671 podStartE2EDuration="2.111482671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:34:46.109931776 +0000 UTC m=+1933.145226618" watchObservedRunningTime="2026-10-31 16:34:46.111482671 +0000 UTC m=+1933.146777490" Oct 31 16:34:46.874764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:46.874729 4880 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 16:34:46.875192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:46.874780 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 16:34:47.093155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:47.093114 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a0d474-ca84-45b8-8711-a944b019383c" path="/var/lib/kubelet/pods/15a0d474-ca84-45b8-8711-a944b019383c/volumes" Oct 31 16:34:47.093690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:47.093671 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4180fbb7-3252-40b1-b401-34b666278d57" path="/var/lib/kubelet/pods/4180fbb7-3252-40b1-b401-34b666278d57/volumes" Oct 31 16:34:47.094123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:47.094108 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" path="/var/lib/kubelet/pods/5a05bcb5-0ec3-42ea-8c14-0e100ea8943b/volumes" Oct 31 16:34:47.094891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:47.094871 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" path="/var/lib/kubelet/pods/deab96e1-8cff-43b4-be2c-6a0c3b64348d/volumes" Oct 31 16:34:47.873073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:47.873039 4880 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 16:34:47.873287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:47.873088 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 16:34:48.590988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:48.590955 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:48.872101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:48.872025 4880 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 16:34:48.872101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:48.872084 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 16:34:49.872518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:49.872477 4880 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 16:34:49.872956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:49.872529 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 16:34:50.746632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:50.746595 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:50.872320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:50.872283 4880 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 31 16:34:50.872473 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:50.872344 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 31 16:34:51.876805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:51.876776 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:51.877314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:51.877054 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:51.877705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:51.877685 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:34:53.486461 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:53.486397 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:34:56.592683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.592650 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf"] Oct 31 16:34:56.593065 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.592837 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" containerID="cri-o://9ff870fa4c5bd4213af3993645752b4f99142070d452bd60a2044a3126d93f09" gracePeriod=30 Oct 31 16:34:56.694753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.694719 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk"] Oct 31 16:34:56.694950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.694928 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" containerID="cri-o://768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f" gracePeriod=30 Oct 31 16:34:56.849484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.849383 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84"] Oct 31 16:34:56.849664 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.849628 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" containerID="cri-o://1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6" gracePeriod=30 Oct 31 16:34:56.864097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.864067 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct"] Oct 31 16:34:56.864323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:56.864284 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" containerID="cri-o://fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b" gracePeriod=30 Oct 31 16:34:58.666201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:58.666169 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:34:58.666614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:58.666230 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:34:58.741562 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:34:58.741532 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:01.204317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:01.204276 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggxdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:35:01.204767 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:01.204338 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:35:01.734663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:01.734504 4880 generic.go:356] "Generic (PLEG): container finished" podID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerID="9ff870fa4c5bd4213af3993645752b4f99142070d452bd60a2044a3126d93f09" exitCode=0 Oct 31 16:35:01.734663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:01.734577 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" event={"ID":"e19ed2ae-c6c2-455f-b835-2b59a9d5f461","Type":"ContainerDied","Data":"9ff870fa4c5bd4213af3993645752b4f99142070d452bd60a2044a3126d93f09"} Oct 31 16:35:02.014794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.014676 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:35:02.018256 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.018235 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:35:02.024218 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024195 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-certs\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024270 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-podinfo\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.024332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024310 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-certs\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.024456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024350 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-socket\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024404 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-kube-api-access-fn52n\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-kube-api-access-fn52n\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.024555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024452 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3460a123-5789-42de-9c62-1b51a177102a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3460a123-5789-42de-9c62-1b51a177102a-istio-podinfo\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024490 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-envoy\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024518 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-token\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-token\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.024702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024567 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istiod-ca-cert\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.024702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024600 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-data\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024638 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3460a123-5789-42de-9c62-1b51a177102a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3460a123-5789-42de-9c62-1b51a177102a-istiod-ca-cert\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024852 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024696 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-istio-token\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-istio-token\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024852 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024729 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-kube-api-access-mx62m\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-kube-api-access-mx62m\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.024852 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024762 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-socket\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.024852 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024843 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-envoy\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.025053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.024904 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-credential-socket\") pod \"3460a123-5789-42de-9c62-1b51a177102a\" (UID: \"3460a123-5789-42de-9c62-1b51a177102a\") " Oct 31 16:35:02.025115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025067 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-data\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.025185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025130 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-credential-socket\") pod \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\" (UID: \"e19ed2ae-c6c2-455f-b835-2b59a9d5f461\") " Oct 31 16:35:02.025651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025372 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3460a123-5789-42de-9c62-1b51a177102a-istiod-ca-cert" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:02.025651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025498 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-credential-socket" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.025651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025539 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-data" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.025651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025573 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-socket" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025654 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-credential-socket" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.025873 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-data" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026018 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-certs" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026106 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-socket" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026299 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026319 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026321 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istiod-ca-cert" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026330 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-certs" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026334 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026361 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3460a123-5789-42de-9c62-1b51a177102a-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026388 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026402 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026414 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.026846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.026427 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.028280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.028242 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-envoy" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.028360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.028318 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-istio-token" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.028401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.028359 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-token" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.028641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.028618 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-podinfo" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:35:02.029282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.029261 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3460a123-5789-42de-9c62-1b51a177102a-istio-podinfo" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:35:02.029847 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.029827 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-envoy" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.029938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.029863 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-kube-api-access-mx62m" pod "3460a123-5789-42de-9c62-1b51a177102a" (UID: "3460a123-5789-42de-9c62-1b51a177102a"). InnerVolumeSpecName "kube-api-access-mx62m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.030000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.029983 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-kube-api-access-fn52n" pod "e19ed2ae-c6c2-455f-b835-2b59a9d5f461" (UID: "e19ed2ae-c6c2-455f-b835-2b59a9d5f461"). InnerVolumeSpecName "kube-api-access-fn52n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.127603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127576 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127602 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127611 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn52n\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-kube-api-access-fn52n\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127621 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3460a123-5789-42de-9c62-1b51a177102a-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127630 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3460a123-5789-42de-9c62-1b51a177102a-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127640 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127651 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127664 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127677 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx62m\" (UniqueName: \"kubernetes.io/projected/3460a123-5789-42de-9c62-1b51a177102a-kube-api-access-mx62m\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.127795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.127690 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e19ed2ae-c6c2-455f-b835-2b59a9d5f461-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.137516 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.137492 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:35:02.191293 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.191260 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229012 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-podinfo\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229584 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istiod-ca-cert\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229622 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-token\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-token\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229674 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-kube-api-access-tgmtl\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-kube-api-access-tgmtl\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229706 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-socket\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229750 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-socket\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229829 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-kube-api-access-9p9w9\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-kube-api-access-9p9w9\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229870 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-envoy\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229908 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-certs\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.229954 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-podinfo\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230017 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istiod-ca-cert\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230063 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-envoy\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.230158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230113 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-credential-socket\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.231172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230193 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-data\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.231172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230273 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-credential-socket\") pod \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\" (UID: \"95f9f219-fdf3-42cc-8ac5-e6f4588649c2\") " Oct 31 16:35:02.231172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230312 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-token\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-token\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.231172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230376 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-data\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.231172 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.230414 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-certs\") pod \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\" (UID: \"3f9251b2-071b-4d53-bdea-3aa2efd0f90c\") " Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.232217 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-credential-socket" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.232501 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istiod-ca-cert" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.232726 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-credential-socket" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.232980 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-certs" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.233205 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-data" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.233450 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-socket" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.233715 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-socket" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.234035 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istiod-ca-cert" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.234517 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-certs" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.235762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.234944 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-data" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.236328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.236179 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-token" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.236778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.236752 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-token" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.237602 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.237470 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-podinfo" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:35:02.237602 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.237516 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-envoy" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.237602 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.237580 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-podinfo" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:35:02.238113 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.238096 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-kube-api-access-tgmtl" pod "95f9f219-fdf3-42cc-8ac5-e6f4588649c2" (UID: "95f9f219-fdf3-42cc-8ac5-e6f4588649c2"). InnerVolumeSpecName "kube-api-access-tgmtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.239108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.239081 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-kube-api-access-9p9w9" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "kube-api-access-9p9w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.239421 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.239402 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-envoy" pod "3f9251b2-071b-4d53-bdea-3aa2efd0f90c" (UID: "3f9251b2-071b-4d53-bdea-3aa2efd0f90c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:02.332432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332401 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p9w9\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-kube-api-access-9p9w9\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332429 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332438 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332447 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332457 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332464 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332472 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332480 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332490 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332499 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332506 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332514 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332524 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332532 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332540 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332548 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgmtl\" (UniqueName: \"kubernetes.io/projected/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-kube-api-access-tgmtl\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332556 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95f9f219-fdf3-42cc-8ac5-e6f4588649c2-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.332650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.332563 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9251b2-071b-4d53-bdea-3aa2efd0f90c-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.744382 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.744350 4880 generic.go:356] "Generic (PLEG): container finished" podID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerID="fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b" exitCode=0 Oct 31 16:35:02.744569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.744395 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" Oct 31 16:35:02.744569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.744430 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" event={"ID":"95f9f219-fdf3-42cc-8ac5-e6f4588649c2","Type":"ContainerDied","Data":"fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b"} Oct 31 16:35:02.744569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.744477 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct" event={"ID":"95f9f219-fdf3-42cc-8ac5-e6f4588649c2","Type":"ContainerDied","Data":"a6295638f0dede7b273124ed5e1933310a2701ce0fcda2f225e569a8ae917a2c"} Oct 31 16:35:02.744569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.744502 4880 scope.go:122] "RemoveContainer" containerID="fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b" Oct 31 16:35:02.746095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.746072 4880 generic.go:356] "Generic (PLEG): container finished" podID="3460a123-5789-42de-9c62-1b51a177102a" containerID="768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f" exitCode=0 Oct 31 16:35:02.746194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.746115 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" event={"ID":"3460a123-5789-42de-9c62-1b51a177102a","Type":"ContainerDied","Data":"768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f"} Oct 31 16:35:02.746194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.746125 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" Oct 31 16:35:02.746194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.746176 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk" event={"ID":"3460a123-5789-42de-9c62-1b51a177102a","Type":"ContainerDied","Data":"22a031fc19bfeb9e67af6316fc07d1bfe4f46b7b98a2490520d1977896457ab3"} Oct 31 16:35:02.747812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.747789 4880 generic.go:356] "Generic (PLEG): container finished" podID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerID="1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6" exitCode=0 Oct 31 16:35:02.747940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.747863 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" event={"ID":"3f9251b2-071b-4d53-bdea-3aa2efd0f90c","Type":"ContainerDied","Data":"1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6"} Oct 31 16:35:02.747940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.747904 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" event={"ID":"3f9251b2-071b-4d53-bdea-3aa2efd0f90c","Type":"ContainerDied","Data":"2761639283a6ab742d632b634b6e473fc0ee108a6ca1022649fd3dcf8353ef16"} Oct 31 16:35:02.747940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.747934 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84" Oct 31 16:35:02.749712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.749593 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" Oct 31 16:35:02.749712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.749670 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf" event={"ID":"e19ed2ae-c6c2-455f-b835-2b59a9d5f461","Type":"ContainerDied","Data":"017cada8c6ade42dd3e6f44e952b1547f8eb6ef850499bf60463b8312959d2a3"} Oct 31 16:35:02.757935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.757920 4880 scope.go:122] "RemoveContainer" containerID="fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b" Oct 31 16:35:02.758304 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:35:02.758180 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b\": container with ID starting with fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b not found: ID does not exist" containerID="fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b" Oct 31 16:35:02.758304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.758210 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b"} err="failed to get container status \"fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b\": rpc error: code = NotFound desc = could not find container \"fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b\": container with ID starting with fa6a0b985cb64cbd14382afbff135070a654987d345cdb7d32d6d60873e9798b not found: ID does not exist" Oct 31 16:35:02.758304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.758229 4880 scope.go:122] "RemoveContainer" containerID="768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f" Oct 31 16:35:02.770328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.770313 4880 scope.go:122] "RemoveContainer" containerID="768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f" Oct 31 16:35:02.770574 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:35:02.770558 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f\": container with ID starting with 768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f not found: ID does not exist" containerID="768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f" Oct 31 16:35:02.770622 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.770581 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f"} err="failed to get container status \"768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f\": rpc error: code = NotFound desc = could not find container \"768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f\": container with ID starting with 768623c43f50240f48704233edca74a0ab8eeb84e082d86579c485e75ebc587f not found: ID does not exist" Oct 31 16:35:02.770622 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.770597 4880 scope.go:122] "RemoveContainer" containerID="1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6" Oct 31 16:35:02.780802 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.780785 4880 scope.go:122] "RemoveContainer" containerID="1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6" Oct 31 16:35:02.781066 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:35:02.781044 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6\": container with ID starting with 1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6 not found: ID does not exist" containerID="1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6" Oct 31 16:35:02.781123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.781075 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6"} err="failed to get container status \"1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6\": rpc error: code = NotFound desc = could not find container \"1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6\": container with ID starting with 1d8aa3ab1286bdf3bb761a4273adbb8fb8b9a47486256aa8c5ab209b1848efb6 not found: ID does not exist" Oct 31 16:35:02.781123 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.781092 4880 scope.go:122] "RemoveContainer" containerID="9ff870fa4c5bd4213af3993645752b4f99142070d452bd60a2044a3126d93f09" Oct 31 16:35:02.899032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.899006 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk"] Oct 31 16:35:02.945159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:02.945117 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-rw2sk"] Oct 31 16:35:03.006034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.005997 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct"] Oct 31 16:35:03.010341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.010315 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-c6nct"] Oct 31 16:35:03.092560 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.092524 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3460a123-5789-42de-9c62-1b51a177102a" path="/var/lib/kubelet/pods/3460a123-5789-42de-9c62-1b51a177102a/volumes" Oct 31 16:35:03.093118 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.093097 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" path="/var/lib/kubelet/pods/95f9f219-fdf3-42cc-8ac5-e6f4588649c2/volumes" Oct 31 16:35:03.110403 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.110374 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf"] Oct 31 16:35:03.114684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.114662 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggxdf"] Oct 31 16:35:03.214858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.214771 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84"] Oct 31 16:35:03.218302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.218269 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-hqc84"] Oct 31 16:35:03.881953 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:35:03.881913 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:35:03.949498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:03.949463 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:05.092644 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:05.092608 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" path="/var/lib/kubelet/pods/3f9251b2-071b-4d53-bdea-3aa2efd0f90c/volumes" Oct 31 16:35:05.093179 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:05.093163 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" path="/var/lib/kubelet/pods/e19ed2ae-c6c2-455f-b835-2b59a9d5f461/volumes" Oct 31 16:35:05.990171 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:05.990124 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:07.000423 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:07.000384 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:07.144478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:07.144439 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:09.066244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:09.066212 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:11.358701 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:11.358653 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:11.499885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:11.499853 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:13.953459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:13.953418 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:16.106124 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:16.106087 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:16.293001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:16.292960 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:19.252740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:19.252703 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:24.139182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:24.137199 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:25.742879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:25.742843 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:26.022351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:26.022266 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:26.175695 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:26.175663 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:26.872078 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:26.872039 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:28.661047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:28.661011 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv"] Oct 31 16:35:29.015504 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:29.015424 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:29.153663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:29.153633 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:29.458343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:29.458297 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:31.295308 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:31.295264 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:32.561420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:32.561381 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:32.735389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:32.734673 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:32.894212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:32.894173 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:33.662272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:33.662238 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" podUID="18fd4ac2-abea-482c-a786-4e75e5d06b64" containerName="payload-processing" containerID="cri-o://2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb" gracePeriod=25 Oct 31 16:35:33.893915 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:35:33.893851 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:35:33.946234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:33.946211 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:35:34.007091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.007056 4880 generic.go:356] "Generic (PLEG): container finished" podID="18fd4ac2-abea-482c-a786-4e75e5d06b64" containerID="2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb" exitCode=0 Oct 31 16:35:34.007287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.007105 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" Oct 31 16:35:34.007287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.007148 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" event={"ID":"18fd4ac2-abea-482c-a786-4e75e5d06b64","Type":"ContainerDied","Data":"2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb"} Oct 31 16:35:34.007287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.007241 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv" event={"ID":"18fd4ac2-abea-482c-a786-4e75e5d06b64","Type":"ContainerDied","Data":"265b5a40eb43f524ff1fa71eb8c636f64bad0a6a757a92fe27d2532421b72e4a"} Oct 31 16:35:34.007287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.007271 4880 scope.go:122] "RemoveContainer" containerID="2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb" Oct 31 16:35:34.018216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.018194 4880 scope.go:122] "RemoveContainer" containerID="2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb" Oct 31 16:35:34.018494 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:35:34.018475 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb\": container with ID starting with 2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb not found: ID does not exist" containerID="2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb" Oct 31 16:35:34.018565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.018506 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb"} err="failed to get container status \"2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb\": rpc error: code = NotFound desc = could not find container \"2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb\": container with ID starting with 2b2660c7ea6c13e47c3b7a96ffc6cb17485bad82752a7ca8b0d7eba5936cbbdb not found: ID does not exist" Oct 31 16:35:34.072918 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.072882 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18fd4ac2-abea-482c-a786-4e75e5d06b64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd4ac2-abea-482c-a786-4e75e5d06b64-plugins-config-volume\") pod \"18fd4ac2-abea-482c-a786-4e75e5d06b64\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " Oct 31 16:35:34.073175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.073128 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fd4ac2-abea-482c-a786-4e75e5d06b64-kube-api-access-wxdxl\" (UniqueName: \"kubernetes.io/projected/18fd4ac2-abea-482c-a786-4e75e5d06b64-kube-api-access-wxdxl\") pod \"18fd4ac2-abea-482c-a786-4e75e5d06b64\" (UID: \"18fd4ac2-abea-482c-a786-4e75e5d06b64\") " Oct 31 16:35:34.073324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.073301 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fd4ac2-abea-482c-a786-4e75e5d06b64-plugins-config-volume" pod "18fd4ac2-abea-482c-a786-4e75e5d06b64" (UID: "18fd4ac2-abea-482c-a786-4e75e5d06b64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:34.073902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.073877 4880 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd4ac2-abea-482c-a786-4e75e5d06b64-plugins-config-volume\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:34.075263 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.075236 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fd4ac2-abea-482c-a786-4e75e5d06b64-kube-api-access-wxdxl" pod "18fd4ac2-abea-482c-a786-4e75e5d06b64" (UID: "18fd4ac2-abea-482c-a786-4e75e5d06b64"). InnerVolumeSpecName "kube-api-access-wxdxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:34.081387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.081353 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:34.175370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.175275 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxdxl\" (UniqueName: \"kubernetes.io/projected/18fd4ac2-abea-482c-a786-4e75e5d06b64-kube-api-access-wxdxl\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:34.271180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.271108 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:34.421538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.421497 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv"] Oct 31 16:35:34.425483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:34.425415 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdc828-647975f48f-wvqlv"] Oct 31 16:35:35.003742 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:35.003701 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:35.093519 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:35.093486 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18fd4ac2-abea-482c-a786-4e75e5d06b64" path="/var/lib/kubelet/pods/18fd4ac2-abea-482c-a786-4e75e5d06b64/volumes" Oct 31 16:35:36.134887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:36.134851 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:36.580754 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:36.580664 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:36.726897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:36.726864 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:39.898248 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:39.898209 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:40.111840 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:40.111802 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:40.262925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:40.262815 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:41.340747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:41.340706 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:43.130401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:43.130364 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:43.318396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:43.318347 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:43.560752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:43.560720 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:43.862264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:43.862162 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:44.485612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:44.485579 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:48.389520 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:48.389484 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:50.241539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:50.241504 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:50.427781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:50.427737 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:50.530245 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:50.530154 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l"] Oct 31 16:35:50.530435 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:50.530404 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" containerID="cri-o://21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9" gracePeriod=30 Oct 31 16:35:50.580899 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:50.580863 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:51.879326 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:51.879285 4880 patch_prober.go:29] interesting pod/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:35:51.879701 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:51.879353 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:35:52.654187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:52.654116 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:52.975628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:52.975522 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:53.305751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:53.305647 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:53.837245 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:53.837208 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:54.693096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:54.693055 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:54.895110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:54.895069 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:55.055185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.055079 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:55.822129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.822071 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:35:55.961584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961546 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-data\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.961772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961606 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-socket\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.961772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961632 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-podinfo\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.961772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961700 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-token\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-token\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.961772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961732 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-envoy\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.962000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961799 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-credential-socket\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.962000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961887 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-data" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:55.962000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.961959 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-certs\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.962203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.962007 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5585ac35-f2e6-4344-9d72-92ab372d5918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5585ac35-f2e6-4344-9d72-92ab372d5918-istiod-ca-cert\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.962203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.962038 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-kube-api-access-dt8xh\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-kube-api-access-dt8xh\") pod \"5585ac35-f2e6-4344-9d72-92ab372d5918\" (UID: \"5585ac35-f2e6-4344-9d72-92ab372d5918\") " Oct 31 16:35:55.962406 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.962360 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-socket" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:55.962891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.962838 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:55.962891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.962865 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:55.963064 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.962959 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5585ac35-f2e6-4344-9d72-92ab372d5918-istiod-ca-cert" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:55.963064 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.962987 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-credential-socket" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:55.963064 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.963027 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-certs" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:55.965468 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.965435 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-envoy" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:35:55.965582 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.965492 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-token" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:55.965582 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.965504 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-podinfo" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:35:55.965705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:55.965583 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-kube-api-access-dt8xh" pod "5585ac35-f2e6-4344-9d72-92ab372d5918" (UID: "5585ac35-f2e6-4344-9d72-92ab372d5918"). InnerVolumeSpecName "kube-api-access-dt8xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:56.064385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.064353 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:56.064385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.064381 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:56.064385 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.064393 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5585ac35-f2e6-4344-9d72-92ab372d5918-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:56.064617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.064403 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5585ac35-f2e6-4344-9d72-92ab372d5918-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:56.064617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.064414 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt8xh\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-kube-api-access-dt8xh\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:56.064617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.064423 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:56.064617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.064433 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5585ac35-f2e6-4344-9d72-92ab372d5918-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:35:56.250332 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.250297 4880 generic.go:356] "Generic (PLEG): container finished" podID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerID="21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9" exitCode=0 Oct 31 16:35:56.250529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.250358 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" Oct 31 16:35:56.250529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.250366 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" event={"ID":"5585ac35-f2e6-4344-9d72-92ab372d5918","Type":"ContainerDied","Data":"21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9"} Oct 31 16:35:56.250529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.250487 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l" event={"ID":"5585ac35-f2e6-4344-9d72-92ab372d5918","Type":"ContainerDied","Data":"84e7931af2498c7cac0801116fc9f565cb9a156266bab88c5f7b2a5d6e74b23a"} Oct 31 16:35:56.250529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.250520 4880 scope.go:122] "RemoveContainer" containerID="21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9" Oct 31 16:35:56.262193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.262167 4880 scope.go:122] "RemoveContainer" containerID="21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9" Oct 31 16:35:56.262450 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:35:56.262424 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9\": container with ID starting with 21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9 not found: ID does not exist" containerID="21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9" Oct 31 16:35:56.262509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.262462 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9"} err="failed to get container status \"21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9\": rpc error: code = NotFound desc = could not find container \"21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9\": container with ID starting with 21ade3f338a076d4bb64b47c591a15d1ea4c7e81f6b89ada3aab16819dc715e9 not found: ID does not exist" Oct 31 16:35:56.408706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.408667 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l"] Oct 31 16:35:56.412674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:56.412649 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bdc828-openshift-default-569d5846c-jhr9l"] Oct 31 16:35:57.093017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:57.092974 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" path="/var/lib/kubelet/pods/5585ac35-f2e6-4344-9d72-92ab372d5918/volumes" Oct 31 16:35:57.104984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:57.104955 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:57.289720 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:57.289685 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:57.442669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:57.442629 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:35:58.908490 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:35:58.908455 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:00.141203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:00.141172 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:00.329246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:00.329207 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:00.472645 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:00.472563 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:02.408149 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:02.408103 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:02.633119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:02.633080 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:03.374336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:03.374297 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:03.522406 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:03.522362 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:03.564214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:03.564170 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l"] Oct 31 16:36:03.667184 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:03.667147 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr"] Oct 31 16:36:03.888796 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:36:03.888758 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:36:03.927052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:03.926970 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:04.077602 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:04.077568 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:05.303343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:05.303301 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:07.219532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:07.219483 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:07.406366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:07.406324 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:08.565531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:08.565487 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" podUID="888e0fd6-1a89-41e5-a51b-4c506c1f9399" containerName="payload-processing" containerID="cri-o://f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8" gracePeriod=25 Oct 31 16:36:08.573031 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:36:08.572962 4880 logging.go:55] [core] [Channel #442 SubChannel #444]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.118:50052", ServerName: "10.132.0.118:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.118:50052: connect: connection refused" Oct 31 16:36:08.667970 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:08.667929 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" podUID="8c1cf374-95c4-47e7-9ed3-5b38110a9c60" containerName="payload-pre-processing" containerID="cri-o://c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883" gracePeriod=25 Oct 31 16:36:08.885538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:08.885509 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:36:08.953527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:08.953499 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:36:09.022003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.021974 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-plugins-config-volume\") pod \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " Oct 31 16:36:09.022208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.022039 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-kube-api-access-8qbx2\" (UniqueName: \"kubernetes.io/projected/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-kube-api-access-8qbx2\") pod \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\" (UID: \"8c1cf374-95c4-47e7-9ed3-5b38110a9c60\") " Oct 31 16:36:09.022208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.022076 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888e0fd6-1a89-41e5-a51b-4c506c1f9399-kube-api-access-9x896\" (UniqueName: \"kubernetes.io/projected/888e0fd6-1a89-41e5-a51b-4c506c1f9399-kube-api-access-9x896\") pod \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " Oct 31 16:36:09.022208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.022171 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888e0fd6-1a89-41e5-a51b-4c506c1f9399-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888e0fd6-1a89-41e5-a51b-4c506c1f9399-plugins-config-volume\") pod \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\" (UID: \"888e0fd6-1a89-41e5-a51b-4c506c1f9399\") " Oct 31 16:36:09.022460 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.022433 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-plugins-config-volume" pod "8c1cf374-95c4-47e7-9ed3-5b38110a9c60" (UID: "8c1cf374-95c4-47e7-9ed3-5b38110a9c60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:36:09.022662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.022631 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888e0fd6-1a89-41e5-a51b-4c506c1f9399-plugins-config-volume" pod "888e0fd6-1a89-41e5-a51b-4c506c1f9399" (UID: "888e0fd6-1a89-41e5-a51b-4c506c1f9399"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:36:09.023010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.022983 4880 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-plugins-config-volume\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:36:09.023106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.023015 4880 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888e0fd6-1a89-41e5-a51b-4c506c1f9399-plugins-config-volume\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:36:09.024465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.024434 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888e0fd6-1a89-41e5-a51b-4c506c1f9399-kube-api-access-9x896" pod "888e0fd6-1a89-41e5-a51b-4c506c1f9399" (UID: "888e0fd6-1a89-41e5-a51b-4c506c1f9399"). InnerVolumeSpecName "kube-api-access-9x896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:36:09.024541 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.024468 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-kube-api-access-8qbx2" pod "8c1cf374-95c4-47e7-9ed3-5b38110a9c60" (UID: "8c1cf374-95c4-47e7-9ed3-5b38110a9c60"). InnerVolumeSpecName "kube-api-access-8qbx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:36:09.124451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.124423 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qbx2\" (UniqueName: \"kubernetes.io/projected/8c1cf374-95c4-47e7-9ed3-5b38110a9c60-kube-api-access-8qbx2\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:36:09.124451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.124449 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x896\" (UniqueName: \"kubernetes.io/projected/888e0fd6-1a89-41e5-a51b-4c506c1f9399-kube-api-access-9x896\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:36:09.385631 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.385599 4880 generic.go:356] "Generic (PLEG): container finished" podID="8c1cf374-95c4-47e7-9ed3-5b38110a9c60" containerID="c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883" exitCode=0 Oct 31 16:36:09.385797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.385640 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" Oct 31 16:36:09.385797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.385676 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" event={"ID":"8c1cf374-95c4-47e7-9ed3-5b38110a9c60","Type":"ContainerDied","Data":"c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883"} Oct 31 16:36:09.385797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.385734 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr" event={"ID":"8c1cf374-95c4-47e7-9ed3-5b38110a9c60","Type":"ContainerDied","Data":"d9f1ea0498c36412753edddd5ce70d5cff0cc97f2b3271009119f17450a4483c"} Oct 31 16:36:09.385797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.385768 4880 scope.go:122] "RemoveContainer" containerID="c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883" Oct 31 16:36:09.387126 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.387108 4880 generic.go:356] "Generic (PLEG): container finished" podID="888e0fd6-1a89-41e5-a51b-4c506c1f9399" containerID="f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8" exitCode=0 Oct 31 16:36:09.387268 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.387174 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" event={"ID":"888e0fd6-1a89-41e5-a51b-4c506c1f9399","Type":"ContainerDied","Data":"f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8"} Oct 31 16:36:09.387268 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.387229 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" event={"ID":"888e0fd6-1a89-41e5-a51b-4c506c1f9399","Type":"ContainerDied","Data":"17bdc03a1464b13108b18937e50da0caf6031c5bb5ef1eaacf13ccd7e58cfe73"} Oct 31 16:36:09.387268 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.387184 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" Oct 31 16:36:09.396386 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.396358 4880 scope.go:122] "RemoveContainer" containerID="c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883" Oct 31 16:36:09.396639 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:36:09.396615 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883\": container with ID starting with c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883 not found: ID does not exist" containerID="c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883" Oct 31 16:36:09.396688 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.396644 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883"} err="failed to get container status \"c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883\": rpc error: code = NotFound desc = could not find container \"c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883\": container with ID starting with c408cb9820e3a7b7be195eb28c0c44bdd55f4cc104580fe76119f94979de4883 not found: ID does not exist" Oct 31 16:36:09.396688 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.396661 4880 scope.go:122] "RemoveContainer" containerID="f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8" Oct 31 16:36:09.406120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.406105 4880 scope.go:122] "RemoveContainer" containerID="f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8" Oct 31 16:36:09.406412 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:36:09.406392 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8\": container with ID starting with f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8 not found: ID does not exist" containerID="f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8" Oct 31 16:36:09.406484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.406419 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8"} err="failed to get container status \"f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8\": rpc error: code = NotFound desc = could not find container \"f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8\": container with ID starting with f609aff5383a5b1e0a6743846cf2249c6a82ba16a901338dab103bf3978a01b8 not found: ID does not exist" Oct 31 16:36:09.540319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.540289 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l"] Oct 31 16:36:09.572811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.572763 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l" podUID="888e0fd6-1a89-41e5-a51b-4c506c1f9399" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.118:50052\" within 1s: context deadline exceeded" Oct 31 16:36:09.589048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.589021 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a2ae37-7bc5cc6b4c-87d5l"] Oct 31 16:36:09.645621 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.645544 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr"] Oct 31 16:36:09.649612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:09.649586 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a2ae37-744c88897f-hj5sr"] Oct 31 16:36:10.328682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:10.328636 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:11.091556 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:11.091523 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888e0fd6-1a89-41e5-a51b-4c506c1f9399" path="/var/lib/kubelet/pods/888e0fd6-1a89-41e5-a51b-4c506c1f9399/volumes" Oct 31 16:36:11.091947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:11.091932 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1cf374-95c4-47e7-9ed3-5b38110a9c60" path="/var/lib/kubelet/pods/8c1cf374-95c4-47e7-9ed3-5b38110a9c60/volumes" Oct 31 16:36:14.570688 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:14.570564 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:16.113761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:16.113726 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:16.385127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:16.385093 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:16.523628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:16.523590 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:17.027979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:17.027941 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:18.517472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:18.517438 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:18.679291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:18.679244 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:21.403766 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:21.403730 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:22.881010 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:22.880971 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:23.061345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:23.061312 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:23.216722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:23.216675 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:26.411471 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:26.411429 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:30.900447 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:30.900403 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:32.017403 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:32.017363 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:32.407030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:32.406992 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:32.701729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:32.701642 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:33.314555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:33.314523 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:33.629006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:33.628912 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:33.881437 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:36:33.881344 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:36:33.913474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:33.913434 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:34.068234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:34.068197 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:34.425569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:34.425529 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:34.596537 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:34.596504 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:36.866006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:36.865966 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f"] Oct 31 16:36:37.129185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:37.129081 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:37.320347 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:37.320314 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:39.061528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:39.061493 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:39.216980 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:39.216944 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:39.379992 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:39.379892 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:39.529837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:39.529803 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:40.582379 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:40.582338 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:40.879688 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:40.879564 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:41.043861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:41.040933 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:41.866809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:41.866754 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" podUID="73696f60-ddbd-43cf-9972-77497f586709" containerName="payload-pre-processing" containerID="cri-o://5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9" gracePeriod=25 Oct 31 16:36:42.149694 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.149669 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:36:42.207654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.207621 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73696f60-ddbd-43cf-9972-77497f586709-kube-api-access-vzwth\" (UniqueName: \"kubernetes.io/projected/73696f60-ddbd-43cf-9972-77497f586709-kube-api-access-vzwth\") pod \"73696f60-ddbd-43cf-9972-77497f586709\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " Oct 31 16:36:42.207843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.207771 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73696f60-ddbd-43cf-9972-77497f586709-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73696f60-ddbd-43cf-9972-77497f586709-plugins-config-volume\") pod \"73696f60-ddbd-43cf-9972-77497f586709\" (UID: \"73696f60-ddbd-43cf-9972-77497f586709\") " Oct 31 16:36:42.208125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.208101 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73696f60-ddbd-43cf-9972-77497f586709-plugins-config-volume" pod "73696f60-ddbd-43cf-9972-77497f586709" (UID: "73696f60-ddbd-43cf-9972-77497f586709"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:36:42.208404 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.208386 4880 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73696f60-ddbd-43cf-9972-77497f586709-plugins-config-volume\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:36:42.209731 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.209712 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73696f60-ddbd-43cf-9972-77497f586709-kube-api-access-vzwth" pod "73696f60-ddbd-43cf-9972-77497f586709" (UID: "73696f60-ddbd-43cf-9972-77497f586709"). InnerVolumeSpecName "kube-api-access-vzwth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:36:42.309778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.309745 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwth\" (UniqueName: \"kubernetes.io/projected/73696f60-ddbd-43cf-9972-77497f586709-kube-api-access-vzwth\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:36:42.663843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.663813 4880 generic.go:356] "Generic (PLEG): container finished" podID="73696f60-ddbd-43cf-9972-77497f586709" containerID="5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9" exitCode=0 Oct 31 16:36:42.664021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.663870 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" Oct 31 16:36:42.664021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.663883 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" event={"ID":"73696f60-ddbd-43cf-9972-77497f586709","Type":"ContainerDied","Data":"5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9"} Oct 31 16:36:42.664021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.663942 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f" event={"ID":"73696f60-ddbd-43cf-9972-77497f586709","Type":"ContainerDied","Data":"59aed8864a6f2e78cbc3995eb6de00061426f3dbf3e3a227defb58387e15cd8d"} Oct 31 16:36:42.664021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.663969 4880 scope.go:122] "RemoveContainer" containerID="5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9" Oct 31 16:36:42.675712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.675689 4880 scope.go:122] "RemoveContainer" containerID="5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9" Oct 31 16:36:42.675992 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:36:42.675970 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9\": container with ID starting with 5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9 not found: ID does not exist" containerID="5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9" Oct 31 16:36:42.676084 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.676001 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9"} err="failed to get container status \"5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9\": rpc error: code = NotFound desc = could not find container \"5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9\": container with ID starting with 5a9214bc5c92a756a7b5cd8fa6d2eef179410f55a038e102e1cb361ac19936e9 not found: ID does not exist" Oct 31 16:36:42.818640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.818605 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f"] Oct 31 16:36:42.864619 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:42.864578 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc67dd-d48ffc479-shb7f"] Oct 31 16:36:43.092776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:43.092703 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73696f60-ddbd-43cf-9972-77497f586709" path="/var/lib/kubelet/pods/73696f60-ddbd-43cf-9972-77497f586709/volumes" Oct 31 16:36:44.445616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:44.445580 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:49.333373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:49.333335 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:49.747695 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:49.747662 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:51.983275 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:51.983235 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:52.173317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:52.173276 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:54.285823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:54.285785 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:54.479644 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:54.479598 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:36:54.981657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:36:54.981618 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:00.179785 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:00.179750 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:03.880922 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:37:03.880889 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:37:04.873706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:04.873669 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:05.070675 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:05.070642 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:09.932585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:09.932547 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:12.499115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:12.499075 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:12.646151 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:12.646098 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:14.070961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.070920 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654"] Oct 31 16:37:14.071499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.071104 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" podUID="dcedb53b-0e22-45a2-86cd-145c9958cac3" containerName="maas-api" containerID="cri-o://87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78" gracePeriod=30 Oct 31 16:37:14.190312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.190124 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:14.412940 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.412913 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:37:14.478351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.477039 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls\") pod \"dcedb53b-0e22-45a2-86cd-145c9958cac3\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " Oct 31 16:37:14.478351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.477156 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcedb53b-0e22-45a2-86cd-145c9958cac3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcedb53b-0e22-45a2-86cd-145c9958cac3-openshift-service-ca\") pod \"dcedb53b-0e22-45a2-86cd-145c9958cac3\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " Oct 31 16:37:14.478351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.477270 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcedb53b-0e22-45a2-86cd-145c9958cac3-kube-api-access-88gbd\" (UniqueName: \"kubernetes.io/projected/dcedb53b-0e22-45a2-86cd-145c9958cac3-kube-api-access-88gbd\") pod \"dcedb53b-0e22-45a2-86cd-145c9958cac3\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " Oct 31 16:37:14.478351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.477301 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-metrics-certs\") pod \"dcedb53b-0e22-45a2-86cd-145c9958cac3\" (UID: \"dcedb53b-0e22-45a2-86cd-145c9958cac3\") " Oct 31 16:37:14.480694 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.480450 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcedb53b-0e22-45a2-86cd-145c9958cac3-openshift-service-ca" pod "dcedb53b-0e22-45a2-86cd-145c9958cac3" (UID: "dcedb53b-0e22-45a2-86cd-145c9958cac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:14.492538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.491789 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-metrics-certs" pod "dcedb53b-0e22-45a2-86cd-145c9958cac3" (UID: "dcedb53b-0e22-45a2-86cd-145c9958cac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:14.544376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.534996 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcedb53b-0e22-45a2-86cd-145c9958cac3-kube-api-access-88gbd" pod "dcedb53b-0e22-45a2-86cd-145c9958cac3" (UID: "dcedb53b-0e22-45a2-86cd-145c9958cac3"). InnerVolumeSpecName "kube-api-access-88gbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:14.544376 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.536272 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls" pod "dcedb53b-0e22-45a2-86cd-145c9958cac3" (UID: "dcedb53b-0e22-45a2-86cd-145c9958cac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:14.584609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.584575 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:14.584609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.584609 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcedb53b-0e22-45a2-86cd-145c9958cac3-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:14.584878 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.584624 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88gbd\" (UniqueName: \"kubernetes.io/projected/dcedb53b-0e22-45a2-86cd-145c9958cac3-kube-api-access-88gbd\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:14.584878 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.584640 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcedb53b-0e22-45a2-86cd-145c9958cac3-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:14.908681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.908650 4880 generic.go:356] "Generic (PLEG): container finished" podID="dcedb53b-0e22-45a2-86cd-145c9958cac3" containerID="87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78" exitCode=0 Oct 31 16:37:14.908874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.908693 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" Oct 31 16:37:14.908874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.908734 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" event={"ID":"dcedb53b-0e22-45a2-86cd-145c9958cac3","Type":"ContainerDied","Data":"87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78"} Oct 31 16:37:14.908874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.908787 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654" event={"ID":"dcedb53b-0e22-45a2-86cd-145c9958cac3","Type":"ContainerDied","Data":"e6f06470d4fe9f9614073bc4bd06498ef86787ad119bcb8896f4c592663ce379"} Oct 31 16:37:14.908874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.908810 4880 scope.go:122] "RemoveContainer" containerID="87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78" Oct 31 16:37:14.919815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.919794 4880 scope.go:122] "RemoveContainer" containerID="87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78" Oct 31 16:37:14.920102 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:37:14.920085 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78\": container with ID starting with 87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78 not found: ID does not exist" containerID="87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78" Oct 31 16:37:14.920193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:14.920111 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78"} err="failed to get container status \"87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78\": rpc error: code = NotFound desc = could not find container \"87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78\": container with ID starting with 87d1ca03d650dc0acc052d3f9c48ddc6017929ea9fc80fc17db5d7bbd4776e78 not found: ID does not exist" Oct 31 16:37:15.064152 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:15.064100 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654"] Oct 31 16:37:15.156773 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:15.156741 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dc67dd-54797cc856-w2654"] Oct 31 16:37:15.754861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:15.754827 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:15.912092 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:15.912053 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:16.135761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:16.135724 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:16.287440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:16.287406 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:16.705996 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:16.705963 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:17.091888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:17.091809 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcedb53b-0e22-45a2-86cd-145c9958cac3" path="/var/lib/kubelet/pods/dcedb53b-0e22-45a2-86cd-145c9958cac3/volumes" Oct 31 16:37:21.197124 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:21.197090 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:22.519076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:22.519038 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:22.691096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:22.691060 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:22.848407 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:22.848328 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:25.543652 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:25.543580 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24"] Oct 31 16:37:25.544039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:25.543768 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" containerID="cri-o://a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4" gracePeriod=30 Oct 31 16:37:26.329156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:26.329082 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:26.522966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:26.522930 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:26.663122 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:26.663085 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:27.828730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:27.828698 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps"] Oct 31 16:37:27.829228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:27.828921 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" containerID="cri-o://ae81d5e40bd17edb02e911a30ae78588c47874d758d1bd5364d0ced9a7a37f69" gracePeriod=30 Oct 31 16:37:30.260696 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.260661 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:37:30.261148 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.260722 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:37:30.814795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.814769 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:37:30.952405 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952372 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-istio-token\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-istio-token\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952469 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6136a561-9fbf-4010-bfe4-40feb1408578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6136a561-9fbf-4010-bfe4-40feb1408578-istiod-ca-cert\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952529 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-kube-api-access-5rzsd\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-kube-api-access-5rzsd\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952552 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-credential-socket\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952763 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952666 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-envoy\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952763 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952704 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-socket\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952763 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952735 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6136a561-9fbf-4010-bfe4-40feb1408578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6136a561-9fbf-4010-bfe4-40feb1408578-istio-podinfo\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952804 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-data\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.952920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952840 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-credential-socket" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:30.952920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952845 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-certs\") pod \"6136a561-9fbf-4010-bfe4-40feb1408578\" (UID: \"6136a561-9fbf-4010-bfe4-40feb1408578\") " Oct 31 16:37:30.953091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.952992 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-socket" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:30.953091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953050 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-certs" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:30.953272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953251 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-data" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:30.953350 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953300 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6136a561-9fbf-4010-bfe4-40feb1408578-istiod-ca-cert" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:30.953658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953633 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:30.953658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953656 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:30.953830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953666 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:30.953830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953675 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:30.953830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.953683 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6136a561-9fbf-4010-bfe4-40feb1408578-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:30.954782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.954757 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6136a561-9fbf-4010-bfe4-40feb1408578-istio-podinfo" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:37:30.954857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.954844 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-istio-token" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:30.955045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.955017 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-envoy" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:30.955130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:30.955056 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-kube-api-access-5rzsd" pod "6136a561-9fbf-4010-bfe4-40feb1408578" (UID: "6136a561-9fbf-4010-bfe4-40feb1408578"). InnerVolumeSpecName "kube-api-access-5rzsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:31.054939 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.054904 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:31.054939 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.054940 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rzsd\" (UniqueName: \"kubernetes.io/projected/6136a561-9fbf-4010-bfe4-40feb1408578-kube-api-access-5rzsd\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:31.055230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.054960 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6136a561-9fbf-4010-bfe4-40feb1408578-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:31.055230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.054975 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6136a561-9fbf-4010-bfe4-40feb1408578-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:31.057676 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.057648 4880 generic.go:356] "Generic (PLEG): container finished" podID="6136a561-9fbf-4010-bfe4-40feb1408578" containerID="a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4" exitCode=0 Oct 31 16:37:31.057817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.057702 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" Oct 31 16:37:31.057817 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.057775 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" event={"ID":"6136a561-9fbf-4010-bfe4-40feb1408578","Type":"ContainerDied","Data":"a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4"} Oct 31 16:37:31.057930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.057847 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24" event={"ID":"6136a561-9fbf-4010-bfe4-40feb1408578","Type":"ContainerDied","Data":"d984dff26cf90506e88d7b78628efa77127f5afb5ac88dc0a8582a7c77bc5539"} Oct 31 16:37:31.057930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.057876 4880 scope.go:122] "RemoveContainer" containerID="a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4" Oct 31 16:37:31.072958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.072935 4880 scope.go:122] "RemoveContainer" containerID="a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4" Oct 31 16:37:31.073284 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:37:31.073265 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4\": container with ID starting with a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4 not found: ID does not exist" containerID="a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4" Oct 31 16:37:31.073358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.073297 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4"} err="failed to get container status \"a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4\": rpc error: code = NotFound desc = could not find container \"a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4\": container with ID starting with a730e8e979eb9510319475426a1bddb31d5b4ab19aaa51844832a3f6c613e1e4 not found: ID does not exist" Oct 31 16:37:31.224410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.224312 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24"] Oct 31 16:37:31.235183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.227770 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-p4z24"] Oct 31 16:37:31.515071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:31.514976 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:33.073887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.073856 4880 generic.go:356] "Generic (PLEG): container finished" podID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerID="ae81d5e40bd17edb02e911a30ae78588c47874d758d1bd5364d0ced9a7a37f69" exitCode=0 Oct 31 16:37:33.074273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.073930 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" event={"ID":"9e2e1add-2456-4d28-a7cf-d1da326a6cc3","Type":"ContainerDied","Data":"ae81d5e40bd17edb02e911a30ae78588c47874d758d1bd5364d0ced9a7a37f69"} Oct 31 16:37:33.091156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.091113 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" path="/var/lib/kubelet/pods/6136a561-9fbf-4010-bfe4-40feb1408578/volumes" Oct 31 16:37:33.097320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.097292 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:37:33.184495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184466 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istiod-ca-cert\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.184665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184522 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-envoy\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.184665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184575 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-token\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.184665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184654 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-socket\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.184786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184710 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-kube-api-access-lwlkz\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-kube-api-access-lwlkz\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.184786 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184754 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-credential-socket\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.184894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184783 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-data\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.184894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184844 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istiod-ca-cert" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:33.185018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.184972 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-certs\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.185082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.185016 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-socket" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:33.185082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.185072 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-podinfo\") pod \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\" (UID: \"9e2e1add-2456-4d28-a7cf-d1da326a6cc3\") " Oct 31 16:37:33.185265 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.185226 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-data" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:33.185330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.185260 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-credential-socket" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:33.185506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.185484 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-certs" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:33.186563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.186002 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.186563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.186025 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.186563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.186039 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.186563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.186054 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.186563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.186067 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.187276 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.187254 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-envoy" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:33.187341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.187284 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-token" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:33.187388 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.187364 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-podinfo" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:37:33.187422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.187406 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-kube-api-access-lwlkz" pod "9e2e1add-2456-4d28-a7cf-d1da326a6cc3" (UID: "9e2e1add-2456-4d28-a7cf-d1da326a6cc3"). InnerVolumeSpecName "kube-api-access-lwlkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:33.288052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.288009 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.288052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.288045 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.288052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.288060 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwlkz\" (UniqueName: \"kubernetes.io/projected/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-kube-api-access-lwlkz\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.288319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.288074 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2e1add-2456-4d28-a7cf-d1da326a6cc3-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.441745 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.441653 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:37:33.446316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.446291 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:37:33.449867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.449842 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:37:33.454912 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.454889 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:37:33.471733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.471704 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:37:33.477117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.477092 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:37:33.694071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:33.693990 4880 scope.go:122] "RemoveContainer" containerID="ae81d5e40bd17edb02e911a30ae78588c47874d758d1bd5364d0ced9a7a37f69" Oct 31 16:37:33.881116 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:37:33.881070 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:37:34.081684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:34.081599 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" Oct 31 16:37:34.082119 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:34.081599 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps" event={"ID":"9e2e1add-2456-4d28-a7cf-d1da326a6cc3","Type":"ContainerDied","Data":"fa95619d27c54c4c4fefc2ee9229bb0b1dc2b547a35ad495b29448a3278c9b9a"} Oct 31 16:37:34.342017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:34.341928 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps"] Oct 31 16:37:34.345790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:34.345757 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-9v9ps"] Oct 31 16:37:34.781829 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:34.781782 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:35.091090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:35.091014 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" path="/var/lib/kubelet/pods/9e2e1add-2456-4d28-a7cf-d1da326a6cc3/volumes" Oct 31 16:37:35.501405 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:35.501369 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:35.652201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:35.652127 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:36.342254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:36.342213 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:41.180908 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:41.180869 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:43.655747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:43.655669 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq"] Oct 31 16:37:43.656205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:43.655867 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" podUID="62919192-2a24-4d1f-a67c-9d68b72f4e55" containerName="discovery" containerID="cri-o://fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66" gracePeriod=30 Oct 31 16:37:43.935016 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:43.934989 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:37:44.042882 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.042852 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-ca-configmap\") pod \"62919192-2a24-4d1f-a67c-9d68b72f4e55\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " Oct 31 16:37:44.043017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.042921 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-cacerts\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-cacerts\") pod \"62919192-2a24-4d1f-a67c-9d68b72f4e55\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " Oct 31 16:37:44.043017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.042967 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62919192-2a24-4d1f-a67c-9d68b72f4e55-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62919192-2a24-4d1f-a67c-9d68b72f4e55-local-certs\") pod \"62919192-2a24-4d1f-a67c-9d68b72f4e55\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " Oct 31 16:37:44.043094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.043055 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-kubeconfig\") pod \"62919192-2a24-4d1f-a67c-9d68b72f4e55\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " Oct 31 16:37:44.043236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.043131 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-kube-api-access-g7q9g\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-kube-api-access-g7q9g\") pod \"62919192-2a24-4d1f-a67c-9d68b72f4e55\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " Oct 31 16:37:44.045196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.045169 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-token\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-token\") pod \"62919192-2a24-4d1f-a67c-9d68b72f4e55\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " Oct 31 16:37:44.045302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.045283 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-dns-cert\") pod \"62919192-2a24-4d1f-a67c-9d68b72f4e55\" (UID: \"62919192-2a24-4d1f-a67c-9d68b72f4e55\") " Oct 31 16:37:44.045557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.045531 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-ca-configmap" pod "62919192-2a24-4d1f-a67c-9d68b72f4e55" (UID: "62919192-2a24-4d1f-a67c-9d68b72f4e55"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:44.045660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.045635 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62919192-2a24-4d1f-a67c-9d68b72f4e55-local-certs" pod "62919192-2a24-4d1f-a67c-9d68b72f4e55" (UID: "62919192-2a24-4d1f-a67c-9d68b72f4e55"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:37:44.046683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.046397 4880 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-ca-configmap\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.046683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.046422 4880 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/62919192-2a24-4d1f-a67c-9d68b72f4e55-local-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.046683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.046476 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-kube-api-access-g7q9g" pod "62919192-2a24-4d1f-a67c-9d68b72f4e55" (UID: "62919192-2a24-4d1f-a67c-9d68b72f4e55"). InnerVolumeSpecName "kube-api-access-g7q9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:44.047660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.047638 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-kubeconfig" pod "62919192-2a24-4d1f-a67c-9d68b72f4e55" (UID: "62919192-2a24-4d1f-a67c-9d68b72f4e55"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:44.047809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.047780 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-cacerts" pod "62919192-2a24-4d1f-a67c-9d68b72f4e55" (UID: "62919192-2a24-4d1f-a67c-9d68b72f4e55"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:44.048848 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.048823 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-token" pod "62919192-2a24-4d1f-a67c-9d68b72f4e55" (UID: "62919192-2a24-4d1f-a67c-9d68b72f4e55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:44.049533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.048708 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-dns-cert" pod "62919192-2a24-4d1f-a67c-9d68b72f4e55" (UID: "62919192-2a24-4d1f-a67c-9d68b72f4e55"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:44.147365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.147330 4880 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-csr-dns-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.147365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.147364 4880 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-cacerts\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.147550 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.147379 4880 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-kubeconfig\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.147550 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.147395 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7q9g\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-kube-api-access-g7q9g\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.147550 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.147410 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62919192-2a24-4d1f-a67c-9d68b72f4e55-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.149963 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.149936 4880 generic.go:356] "Generic (PLEG): container finished" podID="62919192-2a24-4d1f-a67c-9d68b72f4e55" containerID="fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66" exitCode=0 Oct 31 16:37:44.150089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.149981 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" Oct 31 16:37:44.150089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.149986 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" event={"ID":"62919192-2a24-4d1f-a67c-9d68b72f4e55","Type":"ContainerDied","Data":"fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66"} Oct 31 16:37:44.150089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.150033 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq" event={"ID":"62919192-2a24-4d1f-a67c-9d68b72f4e55","Type":"ContainerDied","Data":"b52dc8f301b131adc5222c6c90ca03a4befca2018e1a0d4dc99bbdbf1115579c"} Oct 31 16:37:44.150089 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.150052 4880 scope.go:122] "RemoveContainer" containerID="fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66" Oct 31 16:37:44.166110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.166089 4880 scope.go:122] "RemoveContainer" containerID="fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66" Oct 31 16:37:44.166443 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:37:44.166426 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66\": container with ID starting with fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66 not found: ID does not exist" containerID="fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66" Oct 31 16:37:44.166515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.166451 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66"} err="failed to get container status \"fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66\": rpc error: code = NotFound desc = could not find container \"fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66\": container with ID starting with fc51ddec02b4426b680da211fa0fd695de79d0a14c1113eea8163f78338acf66 not found: ID does not exist" Oct 31 16:37:44.304947 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.304907 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq"] Oct 31 16:37:44.310193 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:44.310162 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t9zcq"] Oct 31 16:37:45.093060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:45.093021 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62919192-2a24-4d1f-a67c-9d68b72f4e55" path="/var/lib/kubelet/pods/62919192-2a24-4d1f-a67c-9d68b72f4e55/volumes" Oct 31 16:37:46.063814 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:46.063772 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:50.903438 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:50.903373 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:55.612950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:55.612913 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:37:59.811967 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:37:59.811934 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:00.737450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:00.737415 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:00.920058 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:00.920023 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq"] Oct 31 16:38:00.920673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:00.920275 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" podUID="6b5295a8-4227-4d5b-a12a-dfcc39f5c214" containerName="maas-api" containerID="cri-o://5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe" gracePeriod=30 Oct 31 16:38:00.958127 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:00.958091 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:01.130289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.126805 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:01.267972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.267945 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:38:01.328226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.327069 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls\") pod \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " Oct 31 16:38:01.328226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.327150 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-kube-api-access-ccrmh\") pod \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " Oct 31 16:38:01.328226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.327217 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-openshift-service-ca\") pod \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " Oct 31 16:38:01.328226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.327298 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-metrics-certs\") pod \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\" (UID: \"6b5295a8-4227-4d5b-a12a-dfcc39f5c214\") " Oct 31 16:38:01.333472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.333439 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-openshift-service-ca" pod "6b5295a8-4227-4d5b-a12a-dfcc39f5c214" (UID: "6b5295a8-4227-4d5b-a12a-dfcc39f5c214"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:01.336367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.336339 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls" pod "6b5295a8-4227-4d5b-a12a-dfcc39f5c214" (UID: "6b5295a8-4227-4d5b-a12a-dfcc39f5c214"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:01.350636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.350584 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-kube-api-access-ccrmh" pod "6b5295a8-4227-4d5b-a12a-dfcc39f5c214" (UID: "6b5295a8-4227-4d5b-a12a-dfcc39f5c214"). InnerVolumeSpecName "kube-api-access-ccrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:01.353606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.353571 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-metrics-certs" pod "6b5295a8-4227-4d5b-a12a-dfcc39f5c214" (UID: "6b5295a8-4227-4d5b-a12a-dfcc39f5c214"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:01.356205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.356045 4880 generic.go:356] "Generic (PLEG): container finished" podID="6b5295a8-4227-4d5b-a12a-dfcc39f5c214" containerID="5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe" exitCode=0 Oct 31 16:38:01.356205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.356118 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" event={"ID":"6b5295a8-4227-4d5b-a12a-dfcc39f5c214","Type":"ContainerDied","Data":"5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe"} Oct 31 16:38:01.356205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.356183 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" event={"ID":"6b5295a8-4227-4d5b-a12a-dfcc39f5c214","Type":"ContainerDied","Data":"bd51cad74f3c3448ca12ddce5058d8ddf735af96680d07466fe50dda096f4317"} Oct 31 16:38:01.356205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.356208 4880 scope.go:122] "RemoveContainer" containerID="5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe" Oct 31 16:38:01.356480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.356331 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq" Oct 31 16:38:01.371828 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.371811 4880 scope.go:122] "RemoveContainer" containerID="5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe" Oct 31 16:38:01.372097 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:01.372076 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe\": container with ID starting with 5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe not found: ID does not exist" containerID="5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe" Oct 31 16:38:01.372201 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.372104 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe"} err="failed to get container status \"5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe\": rpc error: code = NotFound desc = could not find container \"5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe\": container with ID starting with 5d71b0da521d96c37e12cb768c8a2ee3a659943d7ff654f7eaa86e8eb0a9c7fe not found: ID does not exist" Oct 31 16:38:01.386584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.386492 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:01.428920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.428884 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:01.428920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.428912 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:01.428920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.428922 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:01.428920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.428930 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrmh\" (UniqueName: \"kubernetes.io/projected/6b5295a8-4227-4d5b-a12a-dfcc39f5c214-kube-api-access-ccrmh\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:01.501961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.501931 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq"] Oct 31 16:38:01.505732 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.505711 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-487213-76bbc87bd4-2z4zq"] Oct 31 16:38:01.665968 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:01.665931 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:02.276639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:02.276606 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:02.906495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:02.906459 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:03.100262 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:03.100224 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5295a8-4227-4d5b-a12a-dfcc39f5c214" path="/var/lib/kubelet/pods/6b5295a8-4227-4d5b-a12a-dfcc39f5c214/volumes" Oct 31 16:38:03.231833 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:03.231739 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:03.509653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:03.509561 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:03.882663 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:03.882543 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:38:04.145018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:04.144982 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:06.354948 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.354914 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv"] Oct 31 16:38:06.355584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355557 4880 cpu_manager.go:414] "containerMap: removing container" podUID="6b5295a8-4227-4d5b-a12a-dfcc39f5c214" containerName="maas-api" Oct 31 16:38:06.355584 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355583 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5295a8-4227-4d5b-a12a-dfcc39f5c214" containerName="maas-api" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355598 4880 cpu_manager.go:414] "containerMap: removing container" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355604 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355621 4880 cpu_manager.go:414] "containerMap: removing container" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355629 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355643 4880 cpu_manager.go:414] "containerMap: removing container" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355649 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355658 4880 cpu_manager.go:414] "containerMap: removing container" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355663 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355671 4880 cpu_manager.go:414] "containerMap: removing container" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355675 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355683 4880 cpu_manager.go:414] "containerMap: removing container" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355687 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355698 4880 cpu_manager.go:414] "containerMap: removing container" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355706 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355717 4880 cpu_manager.go:414] "containerMap: removing container" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355722 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355729 4880 cpu_manager.go:414] "containerMap: removing container" podUID="8c1cf374-95c4-47e7-9ed3-5b38110a9c60" containerName="payload-pre-processing" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355734 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1cf374-95c4-47e7-9ed3-5b38110a9c60" containerName="payload-pre-processing" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355749 4880 cpu_manager.go:414] "containerMap: removing container" podUID="dcedb53b-0e22-45a2-86cd-145c9958cac3" containerName="maas-api" Oct 31 16:38:06.355756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355753 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcedb53b-0e22-45a2-86cd-145c9958cac3" containerName="maas-api" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355769 4880 cpu_manager.go:414] "containerMap: removing container" podUID="73696f60-ddbd-43cf-9972-77497f586709" containerName="payload-pre-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355777 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73696f60-ddbd-43cf-9972-77497f586709" containerName="payload-pre-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355789 4880 cpu_manager.go:414] "containerMap: removing container" podUID="888e0fd6-1a89-41e5-a51b-4c506c1f9399" containerName="payload-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355794 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888e0fd6-1a89-41e5-a51b-4c506c1f9399" containerName="payload-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355800 4880 cpu_manager.go:414] "containerMap: removing container" podUID="18fd4ac2-abea-482c-a786-4e75e5d06b64" containerName="payload-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355805 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18fd4ac2-abea-482c-a786-4e75e5d06b64" containerName="payload-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355812 4880 cpu_manager.go:414] "containerMap: removing container" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355816 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355823 4880 cpu_manager.go:414] "containerMap: removing container" podUID="62919192-2a24-4d1f-a67c-9d68b72f4e55" containerName="discovery" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355828 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62919192-2a24-4d1f-a67c-9d68b72f4e55" containerName="discovery" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355836 4880 cpu_manager.go:414] "containerMap: removing container" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355841 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355856 4880 cpu_manager.go:414] "containerMap: removing container" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355861 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355976 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="4180fbb7-3252-40b1-b401-34b666278d57" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355986 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9251b2-071b-4d53-bdea-3aa2efd0f90c" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355991 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="5585ac35-f2e6-4344-9d72-92ab372d5918" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.355997 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="e19ed2ae-c6c2-455f-b835-2b59a9d5f461" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356003 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="3460a123-5789-42de-9c62-1b51a177102a" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356010 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a05bcb5-0ec3-42ea-8c14-0e100ea8943b" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356015 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1cf374-95c4-47e7-9ed3-5b38110a9c60" containerName="payload-pre-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356021 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcedb53b-0e22-45a2-86cd-145c9958cac3" containerName="maas-api" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356027 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="73696f60-ddbd-43cf-9972-77497f586709" containerName="payload-pre-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356033 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="888e0fd6-1a89-41e5-a51b-4c506c1f9399" containerName="payload-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356038 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="18fd4ac2-abea-482c-a786-4e75e5d06b64" containerName="payload-processing" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356044 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a0d474-ca84-45b8-8711-a944b019383c" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356049 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="62919192-2a24-4d1f-a67c-9d68b72f4e55" containerName="discovery" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356054 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f9f219-fdf3-42cc-8ac5-e6f4588649c2" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356059 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="6136a561-9fbf-4010-bfe4-40feb1408578" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356066 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5295a8-4227-4d5b-a12a-dfcc39f5c214" containerName="maas-api" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356072 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="deab96e1-8cff-43b4-be2c-6a0c3b64348d" containerName="istio-proxy" Oct 31 16:38:06.356453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.356078 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2e1add-2456-4d28-a7cf-d1da326a6cc3" containerName="istio-proxy" Oct 31 16:38:06.359751 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.359728 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.388554 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388526 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.388697 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388582 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.388697 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388611 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179e2ff2-6196-4d03-bc6d-74397af48f56-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.388840 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388819 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.388880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388848 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.388927 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388917 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf2pd\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-kube-api-access-qf2pd\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.388965 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388943 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.389016 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.388997 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.389057 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.389048 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.407973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.407952 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a1b30a15-openshift-default-dockercfg-cmlh7\"" Oct 31 16:38:06.490464 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490431 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.490658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490479 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.490658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490633 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179e2ff2-6196-4d03-bc6d-74397af48f56-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.490816 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490795 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.490878 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490832 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.490878 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490858 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.490979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490902 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf2pd\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-kube-api-access-qf2pd\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.490979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490927 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.491076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.490978 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.491076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.491056 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.491195 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.491103 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.491420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.491390 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179e2ff2-6196-4d03-bc6d-74397af48f56-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.491632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.491611 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.491704 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.491655 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.493125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.493102 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.493504 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.493487 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.542624 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.542575 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv"] Oct 31 16:38:06.543961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.543931 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.544071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.543944 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf2pd\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-kube-api-access-qf2pd\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:06.673660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:06.673628 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:07.107131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.107032 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv"] Oct 31 16:38:07.116413 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:38:07.116386 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179e2ff2_6196_4d03_bc6d_74397af48f56.slice/crio-edde7aa4e3a0d202487a47beb5f33fff3684480f113c775f49b7a9a270cdf4a9 WatchSource:0}: Error finding container edde7aa4e3a0d202487a47beb5f33fff3684480f113c775f49b7a9a270cdf4a9: Status 404 returned error can't find the container with id edde7aa4e3a0d202487a47beb5f33fff3684480f113c775f49b7a9a270cdf4a9 Oct 31 16:38:07.118824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.118791 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:07.118897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.118856 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:07.118897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.118883 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:07.401589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.401555 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" event={"ID":"179e2ff2-6196-4d03-bc6d-74397af48f56","Type":"ContainerStarted","Data":"a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0"} Oct 31 16:38:07.402049 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.401610 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" event={"ID":"179e2ff2-6196-4d03-bc6d-74397af48f56","Type":"ContainerStarted","Data":"edde7aa4e3a0d202487a47beb5f33fff3684480f113c775f49b7a9a270cdf4a9"} Oct 31 16:38:07.438197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.438163 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg"] Oct 31 16:38:07.438401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.438376 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" podUID="569fe801-9503-4103-b593-f064e15cf2a6" containerName="maas-api" containerID="cri-o://7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4" gracePeriod=30 Oct 31 16:38:07.512048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.511987 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" podStartSLOduration=1.511955398 podStartE2EDuration="1.511955398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:07.50889122 +0000 UTC m=+2134.544186022" watchObservedRunningTime="2026-10-31 16:38:07.511955398 +0000 UTC m=+2134.547250206" Oct 31 16:38:07.560849 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.560783 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:07.681221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.680215 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:07.782083 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.782050 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:38:07.807421 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.807394 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569fe801-9503-4103-b593-f064e15cf2a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569fe801-9503-4103-b593-f064e15cf2a6-openshift-service-ca\") pod \"569fe801-9503-4103-b593-f064e15cf2a6\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " Oct 31 16:38:07.807574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.807428 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-metrics-certs\") pod \"569fe801-9503-4103-b593-f064e15cf2a6\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " Oct 31 16:38:07.807574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.807509 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569fe801-9503-4103-b593-f064e15cf2a6-kube-api-access-p6zxs\" (UniqueName: \"kubernetes.io/projected/569fe801-9503-4103-b593-f064e15cf2a6-kube-api-access-p6zxs\") pod \"569fe801-9503-4103-b593-f064e15cf2a6\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " Oct 31 16:38:07.807657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.807639 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls\") pod \"569fe801-9503-4103-b593-f064e15cf2a6\" (UID: \"569fe801-9503-4103-b593-f064e15cf2a6\") " Oct 31 16:38:07.807872 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.807842 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569fe801-9503-4103-b593-f064e15cf2a6-openshift-service-ca" pod "569fe801-9503-4103-b593-f064e15cf2a6" (UID: "569fe801-9503-4103-b593-f064e15cf2a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:07.808262 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.808242 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569fe801-9503-4103-b593-f064e15cf2a6-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:07.809774 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.809742 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569fe801-9503-4103-b593-f064e15cf2a6-kube-api-access-p6zxs" pod "569fe801-9503-4103-b593-f064e15cf2a6" (UID: "569fe801-9503-4103-b593-f064e15cf2a6"). InnerVolumeSpecName "kube-api-access-p6zxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:07.809895 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.809829 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-metrics-certs" pod "569fe801-9503-4103-b593-f064e15cf2a6" (UID: "569fe801-9503-4103-b593-f064e15cf2a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:07.810242 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.810218 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls" pod "569fe801-9503-4103-b593-f064e15cf2a6" (UID: "569fe801-9503-4103-b593-f064e15cf2a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:07.912144 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.912100 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:07.912361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.912343 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6zxs\" (UniqueName: \"kubernetes.io/projected/569fe801-9503-4103-b593-f064e15cf2a6-kube-api-access-p6zxs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:07.912500 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:07.912486 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569fe801-9503-4103-b593-f064e15cf2a6-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:08.402187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.402130 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:08.409277 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.409248 4880 generic.go:356] "Generic (PLEG): container finished" podID="569fe801-9503-4103-b593-f064e15cf2a6" containerID="7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4" exitCode=0 Oct 31 16:38:08.409384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.409299 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" Oct 31 16:38:08.409384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.409333 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" event={"ID":"569fe801-9503-4103-b593-f064e15cf2a6","Type":"ContainerDied","Data":"7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4"} Oct 31 16:38:08.409459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.409387 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg" event={"ID":"569fe801-9503-4103-b593-f064e15cf2a6","Type":"ContainerDied","Data":"84a3e40228c45c24da76b63fd9ab5068376c68c5cfa49ade490d25d25d9290f4"} Oct 31 16:38:08.409459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.409404 4880 scope.go:122] "RemoveContainer" containerID="7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4" Oct 31 16:38:08.437827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.437800 4880 scope.go:122] "RemoveContainer" containerID="7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4" Oct 31 16:38:08.438374 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:08.438350 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4\": container with ID starting with 7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4 not found: ID does not exist" containerID="7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4" Oct 31 16:38:08.438533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.438504 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4"} err="failed to get container status \"7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4\": rpc error: code = NotFound desc = could not find container \"7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4\": container with ID starting with 7e329e3758cffaf5cedbd9e7a7b9ed667115c9aa99e8b224a0f99ff77d8abac4 not found: ID does not exist" Oct 31 16:38:08.595824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.595784 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg"] Oct 31 16:38:08.607347 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.604496 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:08.619465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.619436 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d0755f-64dd7cd79d-kj5cg"] Oct 31 16:38:08.691226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.688369 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 31 16:38:08.691226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.688424 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 31 16:38:08.760506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:08.760464 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:09.092881 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:09.092790 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569fe801-9503-4103-b593-f064e15cf2a6" path="/var/lib/kubelet/pods/569fe801-9503-4103-b593-f064e15cf2a6/volumes" Oct 31 16:38:09.393618 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:09.393581 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:09.674572 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:09.674490 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 31 16:38:09.674572 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:09.674557 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 31 16:38:09.687512 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:09.687477 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:09.846162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:09.845682 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:10.197579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:10.197541 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:10.366816 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:10.366772 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:10.540380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:10.540284 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:10.680159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:10.679653 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 31 16:38:10.680159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:10.679710 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 31 16:38:11.498617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:11.498581 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:11.663719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:11.663686 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:11.674069 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:11.674039 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 31 16:38:11.674251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:11.674090 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 31 16:38:12.675017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:12.674982 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 31 16:38:12.675425 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:12.675035 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 31 16:38:13.037640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:13.037553 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:13.214038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:13.213996 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:13.680373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:13.680343 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:13.681078 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:13.681061 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:13.681735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:13.681713 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:38:15.064198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:15.064161 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:16.329157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:16.329095 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:16.534569 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:16.534536 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:16.819039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:16.819002 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:17.634868 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:17.634830 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:17.835170 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:17.835121 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:20.057108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:20.057071 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:21.487270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.487235 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65"] Oct 31 16:38:21.488068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.488053 4880 cpu_manager.go:414] "containerMap: removing container" podUID="569fe801-9503-4103-b593-f064e15cf2a6" containerName="maas-api" Oct 31 16:38:21.488121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.488071 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569fe801-9503-4103-b593-f064e15cf2a6" containerName="maas-api" Oct 31 16:38:21.488272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.488260 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="569fe801-9503-4103-b593-f064e15cf2a6" containerName="maas-api" Oct 31 16:38:21.493997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.493978 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.587821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.587785 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d2bc6c-026d-4006-878c-4697689f8a32-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.587963 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.587837 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpn2\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-kube-api-access-bqpn2\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.588027 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.587980 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.588076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.588063 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.588163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.588128 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d2bc6c-026d-4006-878c-4697689f8a32-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.588285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.588220 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.588285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.588261 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.588285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.588277 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.588422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.588376 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.675748 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.675712 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65"] Oct 31 16:38:21.690920 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.690879 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.691097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.690978 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.691097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.691055 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d2bc6c-026d-4006-878c-4697689f8a32-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.691112 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.691178 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.691204 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.691259 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.691326 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d2bc6c-026d-4006-878c-4697689f8a32-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692006 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.691365 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpn2\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-kube-api-access-bqpn2\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.692453 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.692728 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.692836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.692813 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.693107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.693085 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.693772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.693750 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d2bc6c-026d-4006-878c-4697689f8a32-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.695044 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.695018 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.695538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.695514 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d2bc6c-026d-4006-878c-4697689f8a32-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.744094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.744020 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.744255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.744175 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpn2\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-kube-api-access-bqpn2\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.808897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.808865 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:21.957353 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:21.957320 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb"] Oct 31 16:38:22.213544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.213507 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:22.242375 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.242350 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65"] Oct 31 16:38:22.247732 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:38:22.247701 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d2bc6c_026d_4006_878c_4697689f8a32.slice/crio-25a49c1512c3c2a9da68de89dbf9e692d33b180961cff7edb00a423734238f2f WatchSource:0}: Error finding container 25a49c1512c3c2a9da68de89dbf9e692d33b180961cff7edb00a423734238f2f: Status 404 returned error can't find the container with id 25a49c1512c3c2a9da68de89dbf9e692d33b180961cff7edb00a423734238f2f Oct 31 16:38:22.250693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.250613 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:22.250813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.250750 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:22.250813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.250798 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:22.359269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.359233 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:22.581702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.581596 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" event={"ID":"17d2bc6c-026d-4006-878c-4697689f8a32","Type":"ContainerStarted","Data":"9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d"} Oct 31 16:38:22.581702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.581666 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" event={"ID":"17d2bc6c-026d-4006-878c-4697689f8a32","Type":"ContainerStarted","Data":"25a49c1512c3c2a9da68de89dbf9e692d33b180961cff7edb00a423734238f2f"} Oct 31 16:38:22.695783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.695727 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" podStartSLOduration=1.695714065 podStartE2EDuration="1.695714065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:22.691797477 +0000 UTC m=+2149.727092281" watchObservedRunningTime="2026-10-31 16:38:22.695714065 +0000 UTC m=+2149.731008869" Oct 31 16:38:22.809516 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:22.809483 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:23.811433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:23.811394 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 16:38:23.811876 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:23.811498 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 16:38:24.809903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:24.809863 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 16:38:24.810080 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:24.809916 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 16:38:24.948001 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:24.947965 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:25.809661 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:25.809622 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 16:38:25.809859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:25.809678 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 16:38:26.810050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:26.810014 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 16:38:26.810487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:26.810072 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 16:38:26.957880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:26.957839 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" podUID="739b9172-470f-4212-b53c-3887455786fc" containerName="payload-pre-processing" containerID="cri-o://e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9" gracePeriod=25 Oct 31 16:38:27.264207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.264181 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:38:27.405647 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.405608 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739b9172-470f-4212-b53c-3887455786fc-kube-api-access-sgd76\" (UniqueName: \"kubernetes.io/projected/739b9172-470f-4212-b53c-3887455786fc-kube-api-access-sgd76\") pod \"739b9172-470f-4212-b53c-3887455786fc\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " Oct 31 16:38:27.405831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.405812 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739b9172-470f-4212-b53c-3887455786fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b9172-470f-4212-b53c-3887455786fc-plugins-config-volume\") pod \"739b9172-470f-4212-b53c-3887455786fc\" (UID: \"739b9172-470f-4212-b53c-3887455786fc\") " Oct 31 16:38:27.406294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.406256 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739b9172-470f-4212-b53c-3887455786fc-plugins-config-volume" pod "739b9172-470f-4212-b53c-3887455786fc" (UID: "739b9172-470f-4212-b53c-3887455786fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:27.406486 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.406465 4880 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b9172-470f-4212-b53c-3887455786fc-plugins-config-volume\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:27.407836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.407814 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739b9172-470f-4212-b53c-3887455786fc-kube-api-access-sgd76" pod "739b9172-470f-4212-b53c-3887455786fc" (UID: "739b9172-470f-4212-b53c-3887455786fc"). InnerVolumeSpecName "kube-api-access-sgd76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:27.508275 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.508241 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgd76\" (UniqueName: \"kubernetes.io/projected/739b9172-470f-4212-b53c-3887455786fc-kube-api-access-sgd76\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:27.662870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.662768 4880 generic.go:356] "Generic (PLEG): container finished" podID="739b9172-470f-4212-b53c-3887455786fc" containerID="e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9" exitCode=0 Oct 31 16:38:27.662870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.662821 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" Oct 31 16:38:27.662870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.662848 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" event={"ID":"739b9172-470f-4212-b53c-3887455786fc","Type":"ContainerDied","Data":"e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9"} Oct 31 16:38:27.663175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.662904 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb" event={"ID":"739b9172-470f-4212-b53c-3887455786fc","Type":"ContainerDied","Data":"7a163150d212234c83d4eac1b8c84c6ceb6b6e3d2988e90372e365dea5933170"} Oct 31 16:38:27.663175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.662917 4880 scope.go:122] "RemoveContainer" containerID="e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9" Oct 31 16:38:27.674008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.673984 4880 scope.go:122] "RemoveContainer" containerID="e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9" Oct 31 16:38:27.674266 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:27.674247 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9\": container with ID starting with e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9 not found: ID does not exist" containerID="e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9" Oct 31 16:38:27.674318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.674275 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9"} err="failed to get container status \"e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9\": rpc error: code = NotFound desc = could not find container \"e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9\": container with ID starting with e90c87812a11f29fcbec674f63932c20ee2933e88917a6ff907f0ab86de959d9 not found: ID does not exist" Oct 31 16:38:27.810005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.809967 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Oct 31 16:38:27.810258 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.810023 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Oct 31 16:38:27.817540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.817510 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb"] Oct 31 16:38:27.863989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:27.863958 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3dc1da-5bdd65f95f-hlgpb"] Oct 31 16:38:28.514353 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:28.514314 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:28.814495 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:28.814405 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:28.814964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:28.814761 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:28.815585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:28.815565 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:38:28.825716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:28.825686 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:29.094206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:29.093543 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739b9172-470f-4212-b53c-3887455786fc" path="/var/lib/kubelet/pods/739b9172-470f-4212-b53c-3887455786fc/volumes" Oct 31 16:38:29.932664 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:29.932629 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:30.898525 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:30.898494 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:33.881192 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:33.881148 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:38:33.907152 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:33.907112 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:34.362393 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:34.362350 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:35.027413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:35.027319 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:35.183105 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:35.183058 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:35.332693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:35.332591 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:35.473587 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:35.473545 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:36.592907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.592878 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q"] Oct 31 16:38:36.593611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.593588 4880 cpu_manager.go:414] "containerMap: removing container" podUID="739b9172-470f-4212-b53c-3887455786fc" containerName="payload-pre-processing" Oct 31 16:38:36.593717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.593613 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739b9172-470f-4212-b53c-3887455786fc" containerName="payload-pre-processing" Oct 31 16:38:36.593810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.593798 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="739b9172-470f-4212-b53c-3887455786fc" containerName="payload-pre-processing" Oct 31 16:38:36.598821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.598802 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.653205 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.653168 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q"] Oct 31 16:38:36.681484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681446 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.681693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681538 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.681693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681622 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f02272f-ae28-4942-80c0-0d08040a767e-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.681693 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681675 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46wj\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-kube-api-access-r46wj\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.681865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681724 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.681865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681808 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.681865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681836 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.682025 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681877 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f02272f-ae28-4942-80c0-0d08040a767e-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.682025 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.681912 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.783610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783574 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f02272f-ae28-4942-80c0-0d08040a767e-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.783610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783615 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.783823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783684 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.783823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783728 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.783823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783776 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f02272f-ae28-4942-80c0-0d08040a767e-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.783823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783817 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46wj\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-kube-api-access-r46wj\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783860 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783937 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.783965 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.784243 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-certs\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.784322 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-credential-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.784385 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f02272f-ae28-4942-80c0-0d08040a767e-istiod-ca-cert\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.784458 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-data\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.784532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.784498 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-socket\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.785983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.785957 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-envoy\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.786360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.786342 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f02272f-ae28-4942-80c0-0d08040a767e-istio-podinfo\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.836623 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.836590 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-istio-token\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.836740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.836646 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46wj\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-kube-api-access-r46wj\") pod \"e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.912670 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.912644 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:36.923616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:36.923587 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:37.339278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.339191 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q"] Oct 31 16:38:37.345008 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:38:37.344979 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f02272f_ae28_4942_80c0_0d08040a767e.slice/crio-5121830ea0ec2c8edc1e3266dcec8f03a075acaa2abe5f05e150f43a53766d25 WatchSource:0}: Error finding container 5121830ea0ec2c8edc1e3266dcec8f03a075acaa2abe5f05e150f43a53766d25: Status 404 returned error can't find the container with id 5121830ea0ec2c8edc1e3266dcec8f03a075acaa2abe5f05e150f43a53766d25 Oct 31 16:38:37.347679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.347647 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:37.347770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.347715 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:37.347770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.347742 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:38:37.743394 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.743351 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" event={"ID":"8f02272f-ae28-4942-80c0-0d08040a767e","Type":"ContainerStarted","Data":"b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a"} Oct 31 16:38:37.743798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.743440 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" event={"ID":"8f02272f-ae28-4942-80c0-0d08040a767e","Type":"ContainerStarted","Data":"5121830ea0ec2c8edc1e3266dcec8f03a075acaa2abe5f05e150f43a53766d25"} Oct 31 16:38:37.903717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.903665 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podStartSLOduration=1.9036507299999998 podStartE2EDuration="1.90365073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:37.898223077 +0000 UTC m=+2164.933517885" watchObservedRunningTime="2026-10-31 16:38:37.90365073 +0000 UTC m=+2164.938945533" Oct 31 16:38:37.913716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:37.913690 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:38.914874 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:38.914834 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 16:38:38.915313 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:38.914892 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 16:38:39.146158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:39.146110 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:39.306596 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:39.306508 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:39.469181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:39.468833 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:39.913287 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:39.913247 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 16:38:39.913497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:39.913307 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 16:38:40.439106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:40.439070 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:40.913220 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:40.913186 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 16:38:40.913393 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:40.913240 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 16:38:41.913843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:41.913805 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 16:38:41.914264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:41.913863 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 16:38:42.914938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:42.914897 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 16:38:42.915372 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:42.914959 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 16:38:43.913461 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:43.913426 4880 patch_prober.go:29] interesting pod/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 16:38:43.913628 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:43.913480 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 16:38:44.917866 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:44.917833 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:44.918298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:44.918127 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:44.918806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:44.918786 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:38:44.943310 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:44.943278 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:45.116585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:45.116549 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:45.913217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:45.913184 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:48.193505 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.193474 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7204-0147c6"] Oct 31 16:38:48.198537 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.198514 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:48.246104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.246081 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7204-0147c6"] Oct 31 16:38:48.287677 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.287651 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-szvbq\"" Oct 31 16:38:48.372775 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.372748 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hwx\" (UniqueName: \"kubernetes.io/projected/8f5d8be9-7938-4703-9649-480ef3125f69-kube-api-access-h5hwx\") pod \"test-curl-7204-0147c6\" (UID: \"8f5d8be9-7938-4703-9649-480ef3125f69\") " pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:48.474121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.474037 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hwx\" (UniqueName: \"kubernetes.io/projected/8f5d8be9-7938-4703-9649-480ef3125f69-kube-api-access-h5hwx\") pod \"test-curl-7204-0147c6\" (UID: \"8f5d8be9-7938-4703-9649-480ef3125f69\") " pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:48.526527 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.526495 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hwx\" (UniqueName: \"kubernetes.io/projected/8f5d8be9-7938-4703-9649-480ef3125f69-kube-api-access-h5hwx\") pod \"test-curl-7204-0147c6\" (UID: \"8f5d8be9-7938-4703-9649-480ef3125f69\") " pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:48.811933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:48.811842 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:49.183732 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:49.183696 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7204-0147c6"] Oct 31 16:38:49.189273 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:38:49.189242 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f5d8be9_7938_4703_9649_480ef3125f69.slice/crio-35e139266ce6d62a4c5b2d51bf31807c0b3b9f49b9523f86ceb0fe081dbd9585 WatchSource:0}: Error finding container 35e139266ce6d62a4c5b2d51bf31807c0b3b9f49b9523f86ceb0fe081dbd9585: Status 404 returned error can't find the container with id 35e139266ce6d62a4c5b2d51bf31807c0b3b9f49b9523f86ceb0fe081dbd9585 Oct 31 16:38:49.843456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:49.843423 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7204-0147c6" event={"ID":"8f5d8be9-7938-4703-9649-480ef3125f69","Type":"ContainerStarted","Data":"45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca"} Oct 31 16:38:49.843924 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:49.843466 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7204-0147c6" event={"ID":"8f5d8be9-7938-4703-9649-480ef3125f69","Type":"ContainerStarted","Data":"35e139266ce6d62a4c5b2d51bf31807c0b3b9f49b9523f86ceb0fe081dbd9585"} Oct 31 16:38:49.951477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:49.951423 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7204-0147c6" podStartSLOduration=1.951408377 podStartE2EDuration="1.951408377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:49.948794055 +0000 UTC m=+2176.984088858" watchObservedRunningTime="2026-10-31 16:38:49.951408377 +0000 UTC m=+2176.986703182" Oct 31 16:38:50.200781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:50.200744 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:50.502915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:50.502836 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:51.453155 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:51.452427 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7204-0147c6"] Oct 31 16:38:51.460207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:51.459364 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7204-0147c6"] Oct 31 16:38:51.862018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:51.861927 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7204-0147c6" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" containerName="test-curl-7204-0147c6" containerID="cri-o://45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca" gracePeriod=2 Oct 31 16:38:51.909448 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:51.909408 4880 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7204-0147c6\" is forbidden: User \"system:node:ip-10-0-142-127.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-127.us-west-2.compute.internal' and this object" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:52.136063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:52.136033 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:52.434035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:52.433945 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:52.602396 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:52.602362 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:53.098763 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:53.098728 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:53.136753 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:53.136716 4880 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7204-0147c6\" is forbidden: User \"system:node:ip-10-0-142-127.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-127.us-west-2.compute.internal' and this object" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:54.100960 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.100932 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:54.202517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.202481 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5d8be9-7938-4703-9649-480ef3125f69-kube-api-access-h5hwx\" (UniqueName: \"kubernetes.io/projected/8f5d8be9-7938-4703-9649-480ef3125f69-kube-api-access-h5hwx\") pod \"8f5d8be9-7938-4703-9649-480ef3125f69\" (UID: \"8f5d8be9-7938-4703-9649-480ef3125f69\") " Oct 31 16:38:54.204411 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.204382 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5d8be9-7938-4703-9649-480ef3125f69-kube-api-access-h5hwx" pod "8f5d8be9-7938-4703-9649-480ef3125f69" (UID: "8f5d8be9-7938-4703-9649-480ef3125f69"). InnerVolumeSpecName "kube-api-access-h5hwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:54.304521 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.304487 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hwx\" (UniqueName: \"kubernetes.io/projected/8f5d8be9-7938-4703-9649-480ef3125f69-kube-api-access-h5hwx\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:38:54.884410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.884373 4880 generic.go:356] "Generic (PLEG): container finished" podID="8f5d8be9-7938-4703-9649-480ef3125f69" containerID="45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca" exitCode=137 Oct 31 16:38:54.884610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.884424 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:54.884610 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.884476 4880 scope.go:122] "RemoveContainer" containerID="45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca" Oct 31 16:38:54.896956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.896935 4880 scope.go:122] "RemoveContainer" containerID="45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca" Oct 31 16:38:54.897596 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:54.897563 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca\": container with ID starting with 45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca not found: ID does not exist" containerID="45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca" Oct 31 16:38:54.897705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:54.897608 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca"} err="failed to get container status \"45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca\": rpc error: code = NotFound desc = could not find container \"45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca\": container with ID starting with 45abf9c4be844353dd3a6f921b4b262272995aa4b71ca639e3f81ce5e55756ca not found: ID does not exist" Oct 31 16:38:54.941116 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:38:54.941065 4880 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7204-0147c6\" is forbidden: User \"system:node:ip-10-0-142-127.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-127.us-west-2.compute.internal' and this object" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" pod="opendatahub/test-curl-7204-0147c6" Oct 31 16:38:55.091723 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:55.091690 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" path="/var/lib/kubelet/pods/8f5d8be9-7938-4703-9649-480ef3125f69/volumes" Oct 31 16:38:55.787518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:55.787484 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:57.703480 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:57.703441 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:59.178449 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:59.178411 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:59.351482 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:59.351439 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:59.512547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:59.512446 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:38:59.915790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:38:59.915751 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:02.974937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:02.974901 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:03.881318 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:39:03.881281 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:39:04.866916 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:04.866878 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:04.881775 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:04.881741 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p"] Oct 31 16:39:05.020964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:05.020924 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:05.172752 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:05.172715 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:07.174147 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:07.174100 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:07.318478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:07.318441 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:08.451842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:08.451803 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:12.144845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:12.144802 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:12.342676 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:12.342638 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:13.466915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:13.466828 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:13.554935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:13.554881 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph"] Oct 31 16:39:13.555230 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:13.555179 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" containerID="cri-o://8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308" gracePeriod=30 Oct 31 16:39:17.292727 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:17.292692 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-t7qph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:39:17.293167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:17.292757 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:39:18.264339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.264301 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:18.824037 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.823982 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:39:18.878327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878297 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-token\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-token\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878344 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-socket\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878374 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-credential-socket\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878458 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-podinfo\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878477 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-certs\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878530 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878495 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3059189-8881-400a-9fa4-4cef8b277a8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3059189-8881-400a-9fa4-4cef8b277a8d-istiod-ca-cert\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878603 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-envoy\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878643 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-socket" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:18.878729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878671 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-data\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878729 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878719 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-kube-api-access-4wr9x\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-kube-api-access-4wr9x\") pod \"f3059189-8881-400a-9fa4-4cef8b277a8d\" (UID: \"f3059189-8881-400a-9fa4-4cef8b277a8d\") " Oct 31 16:39:18.878937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878810 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-credential-socket" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:18.878937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878825 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-certs" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:18.878937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.878908 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3059189-8881-400a-9fa4-4cef8b277a8d-istiod-ca-cert" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:39:18.879208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.879181 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-data" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:18.879465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.879437 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.879465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.879466 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.879669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.879480 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.879669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.879494 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3059189-8881-400a-9fa4-4cef8b277a8d-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.879669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.879507 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.880716 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.880690 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-envoy" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:18.880835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.880816 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-podinfo" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:39:18.880910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.880889 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-token" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:18.880950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.880924 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-kube-api-access-4wr9x" pod "f3059189-8881-400a-9fa4-4cef8b277a8d" (UID: "f3059189-8881-400a-9fa4-4cef8b277a8d"). InnerVolumeSpecName "kube-api-access-4wr9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:18.980861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.980829 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.980861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.980855 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wr9x\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-kube-api-access-4wr9x\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.980861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.980868 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:18.981106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:18.980877 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3059189-8881-400a-9fa4-4cef8b277a8d-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:19.076740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.076657 4880 generic.go:356] "Generic (PLEG): container finished" podID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerID="8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308" exitCode=0 Oct 31 16:39:19.076740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.076717 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" Oct 31 16:39:19.076915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.076733 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" event={"ID":"f3059189-8881-400a-9fa4-4cef8b277a8d","Type":"ContainerDied","Data":"8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308"} Oct 31 16:39:19.076915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.076813 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph" event={"ID":"f3059189-8881-400a-9fa4-4cef8b277a8d","Type":"ContainerDied","Data":"2a232e9f29a545a3a4573ef21a9af15fcf1696442fca13a71c2b3e7015e2907e"} Oct 31 16:39:19.076915 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.076836 4880 scope.go:122] "RemoveContainer" containerID="8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308" Oct 31 16:39:19.089002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.088981 4880 scope.go:122] "RemoveContainer" containerID="8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308" Oct 31 16:39:19.089400 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:39:19.089351 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308\": container with ID starting with 8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308 not found: ID does not exist" containerID="8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308" Oct 31 16:39:19.089479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.089408 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308"} err="failed to get container status \"8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308\": rpc error: code = NotFound desc = could not find container \"8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308\": container with ID starting with 8a54ac63ebc0b0e30fb9ce299b4ac24a26e4d89c66cd450a32f88f2dc8b3f308 not found: ID does not exist" Oct 31 16:39:19.247517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.247488 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph"] Oct 31 16:39:19.253637 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.253607 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-t7qph"] Oct 31 16:39:19.906580 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:19.906542 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" containerID="cri-o://1afb3413fe539817ebcd0564cbe38b94c3605517c62623ee2e9fb9d598d2b1f8" gracePeriod=45 Oct 31 16:39:20.091108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.091082 4880 generic.go:356] "Generic (PLEG): container finished" podID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerID="1afb3413fe539817ebcd0564cbe38b94c3605517c62623ee2e9fb9d598d2b1f8" exitCode=0 Oct 31 16:39:20.091322 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.091188 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" event={"ID":"aa80a393-96cf-4c65-8d56-84d6ca0292b8","Type":"ContainerDied","Data":"1afb3413fe539817ebcd0564cbe38b94c3605517c62623ee2e9fb9d598d2b1f8"} Oct 31 16:39:20.193009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.192984 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:39:20.311071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.311037 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-dshm\") pod \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " Oct 31 16:39:20.311282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.311207 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa80a393-96cf-4c65-8d56-84d6ca0292b8-kube-api-access-8grr5\" (UniqueName: \"kubernetes.io/projected/aa80a393-96cf-4c65-8d56-84d6ca0292b8-kube-api-access-8grr5\") pod \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " Oct 31 16:39:20.311282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.311258 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tls-certs\") pod \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " Oct 31 16:39:20.311415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.311333 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-model-cache\") pod \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " Oct 31 16:39:20.311415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.311383 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tmp-dir\") pod \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " Oct 31 16:39:20.311522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.311425 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-home\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-home\") pod \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\" (UID: \"aa80a393-96cf-4c65-8d56-84d6ca0292b8\") " Oct 31 16:39:20.311858 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.311819 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-model-cache" pod "aa80a393-96cf-4c65-8d56-84d6ca0292b8" (UID: "aa80a393-96cf-4c65-8d56-84d6ca0292b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:20.312121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.312097 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:20.312364 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.312331 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tmp-dir" pod "aa80a393-96cf-4c65-8d56-84d6ca0292b8" (UID: "aa80a393-96cf-4c65-8d56-84d6ca0292b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:20.312629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.312520 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-home" pod "aa80a393-96cf-4c65-8d56-84d6ca0292b8" (UID: "aa80a393-96cf-4c65-8d56-84d6ca0292b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:20.316433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.316393 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-dshm" pod "aa80a393-96cf-4c65-8d56-84d6ca0292b8" (UID: "aa80a393-96cf-4c65-8d56-84d6ca0292b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:20.316540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.316423 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa80a393-96cf-4c65-8d56-84d6ca0292b8-kube-api-access-8grr5" pod "aa80a393-96cf-4c65-8d56-84d6ca0292b8" (UID: "aa80a393-96cf-4c65-8d56-84d6ca0292b8"). InnerVolumeSpecName "kube-api-access-8grr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:20.318050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.318021 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tls-certs" pod "aa80a393-96cf-4c65-8d56-84d6ca0292b8" (UID: "aa80a393-96cf-4c65-8d56-84d6ca0292b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:39:20.414295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.414178 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:20.414295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.414210 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:20.414295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.414223 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa80a393-96cf-4c65-8d56-84d6ca0292b8-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:20.414295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.414237 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grr5\" (UniqueName: \"kubernetes.io/projected/aa80a393-96cf-4c65-8d56-84d6ca0292b8-kube-api-access-8grr5\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:20.414295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:20.414248 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa80a393-96cf-4c65-8d56-84d6ca0292b8-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:21.092215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:21.092185 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" path="/var/lib/kubelet/pods/f3059189-8881-400a-9fa4-4cef8b277a8d/volumes" Oct 31 16:39:21.099273 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:21.099254 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" Oct 31 16:39:21.099399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:21.099292 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p" event={"ID":"aa80a393-96cf-4c65-8d56-84d6ca0292b8","Type":"ContainerDied","Data":"966b3ebe99fc2e0dbd97f69d9fa0509615d33b72a7532343c1fd7017b87dfcdf"} Oct 31 16:39:21.099399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:21.099333 4880 scope.go:122] "RemoveContainer" containerID="1afb3413fe539817ebcd0564cbe38b94c3605517c62623ee2e9fb9d598d2b1f8" Oct 31 16:39:21.254673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:21.254639 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p"] Oct 31 16:39:21.297931 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:21.297904 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7f13a/simulator-w2-e7f13a-kserve-69dc6496c4-68f2p"] Oct 31 16:39:22.411399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.411367 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs"] Oct 31 16:39:22.411873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.411589 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" podUID="83c249f9-8a9a-46f7-b502-6cf2d21a2508" containerName="maas-api" containerID="cri-o://8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7" gracePeriod=30 Oct 31 16:39:22.541606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.541572 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:22.745418 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.745391 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:39:22.866156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.865589 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c249f9-8a9a-46f7-b502-6cf2d21a2508-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c249f9-8a9a-46f7-b502-6cf2d21a2508-openshift-service-ca\") pod \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " Oct 31 16:39:22.866156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.865647 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c249f9-8a9a-46f7-b502-6cf2d21a2508-kube-api-access-6jqmh\" (UniqueName: \"kubernetes.io/projected/83c249f9-8a9a-46f7-b502-6cf2d21a2508-kube-api-access-6jqmh\") pod \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " Oct 31 16:39:22.866156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.865812 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-metrics-certs\") pod \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " Oct 31 16:39:22.866156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.865858 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls\") pod \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\" (UID: \"83c249f9-8a9a-46f7-b502-6cf2d21a2508\") " Oct 31 16:39:22.870169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.867535 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c249f9-8a9a-46f7-b502-6cf2d21a2508-openshift-service-ca" pod "83c249f9-8a9a-46f7-b502-6cf2d21a2508" (UID: "83c249f9-8a9a-46f7-b502-6cf2d21a2508"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:39:22.874759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.874024 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls" pod "83c249f9-8a9a-46f7-b502-6cf2d21a2508" (UID: "83c249f9-8a9a-46f7-b502-6cf2d21a2508"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:39:22.874759 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.874622 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-metrics-certs" pod "83c249f9-8a9a-46f7-b502-6cf2d21a2508" (UID: "83c249f9-8a9a-46f7-b502-6cf2d21a2508"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:39:22.876725 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.876689 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c249f9-8a9a-46f7-b502-6cf2d21a2508-kube-api-access-6jqmh" pod "83c249f9-8a9a-46f7-b502-6cf2d21a2508" (UID: "83c249f9-8a9a-46f7-b502-6cf2d21a2508"). InnerVolumeSpecName "kube-api-access-6jqmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:22.968032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.967936 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:22.968032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.967978 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83c249f9-8a9a-46f7-b502-6cf2d21a2508-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:22.968032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.967993 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83c249f9-8a9a-46f7-b502-6cf2d21a2508-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:22.968032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:22.968006 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jqmh\" (UniqueName: \"kubernetes.io/projected/83c249f9-8a9a-46f7-b502-6cf2d21a2508-kube-api-access-6jqmh\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:23.099692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.099653 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" path="/var/lib/kubelet/pods/aa80a393-96cf-4c65-8d56-84d6ca0292b8/volumes" Oct 31 16:39:23.114494 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.114465 4880 generic.go:356] "Generic (PLEG): container finished" podID="83c249f9-8a9a-46f7-b502-6cf2d21a2508" containerID="8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7" exitCode=0 Oct 31 16:39:23.114643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.114522 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" Oct 31 16:39:23.114643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.114521 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" event={"ID":"83c249f9-8a9a-46f7-b502-6cf2d21a2508","Type":"ContainerDied","Data":"8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7"} Oct 31 16:39:23.114643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.114578 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs" event={"ID":"83c249f9-8a9a-46f7-b502-6cf2d21a2508","Type":"ContainerDied","Data":"619dda7d7a49f3a44e167504c60a6a4619ff86801e0a064783ea02bebf07efb0"} Oct 31 16:39:23.114643 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.114607 4880 scope.go:122] "RemoveContainer" containerID="8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7" Oct 31 16:39:23.125372 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.125352 4880 scope.go:122] "RemoveContainer" containerID="8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7" Oct 31 16:39:23.125644 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:39:23.125624 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7\": container with ID starting with 8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7 not found: ID does not exist" containerID="8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7" Oct 31 16:39:23.125690 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.125655 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7"} err="failed to get container status \"8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7\": rpc error: code = NotFound desc = could not find container \"8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7\": container with ID starting with 8475008ac9998985bd488f60123f112c48ad9511fc6e4698c4c501f7ce541fc7 not found: ID does not exist" Oct 31 16:39:23.268029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.267953 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs"] Oct 31 16:39:23.272270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:23.272243 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3dc1da-7c8989f74-t6czs"] Oct 31 16:39:24.076416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:24.076380 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:24.289987 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:24.289951 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:24.434808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:24.434773 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:24.998791 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:24.998753 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:25.091348 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:25.091321 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c249f9-8a9a-46f7-b502-6cf2d21a2508" path="/var/lib/kubelet/pods/83c249f9-8a9a-46f7-b502-6cf2d21a2508/volumes" Oct 31 16:39:28.619478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:28.619419 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5"] Oct 31 16:39:28.619993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:28.619788 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" containerID="cri-o://7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275" gracePeriod=30 Oct 31 16:39:28.724111 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:28.724072 4880 patch_prober.go:29] interesting pod/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:39:28.724296 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:28.724162 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:39:29.500008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:29.499973 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:30.954898 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:30.954859 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:31.109518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:31.109473 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:31.291548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:31.291449 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:31.851029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:31.850983 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:33.882905 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:39:33.882871 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:39:33.911555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:33.911488 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:39:34.017737 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.017694 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-certs\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.017909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.017830 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-kube-api-access-vr5cb\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-kube-api-access-vr5cb\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.017909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.017887 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-envoy\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.017997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.017921 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-credential-socket\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.017997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.017963 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-socket\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.017997 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.017983 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc1fd80-a78f-4942-b879-07baf07ee37a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc1fd80-a78f-4942-b879-07baf07ee37a-istiod-ca-cert\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.018183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018003 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-certs" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:34.018183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018022 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-podinfo\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.018183 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018121 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-token\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-token\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.018354 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018319 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-data\") pod \"ffc1fd80-a78f-4942-b879-07baf07ee37a\" (UID: \"ffc1fd80-a78f-4942-b879-07baf07ee37a\") " Oct 31 16:39:34.018522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018321 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-socket" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:34.018522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018346 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-credential-socket" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:34.018522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018461 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc1fd80-a78f-4942-b879-07baf07ee37a-istiod-ca-cert" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:39:34.018750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.018576 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-data" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:34.019114 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.019093 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.019194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.019122 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.019194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.019153 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffc1fd80-a78f-4942-b879-07baf07ee37a-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.019194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.019170 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.019194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.019184 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.020289 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.020261 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-podinfo" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:39:34.020427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.020404 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-token" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:34.020591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.020566 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-envoy" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:34.020591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.020581 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-kube-api-access-vr5cb" pod "ffc1fd80-a78f-4942-b879-07baf07ee37a" (UID: "ffc1fd80-a78f-4942-b879-07baf07ee37a"). InnerVolumeSpecName "kube-api-access-vr5cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:34.121147 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.121075 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5cb\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-kube-api-access-vr5cb\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.121147 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.121149 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.121384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.121170 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.121384 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.121186 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffc1fd80-a78f-4942-b879-07baf07ee37a-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:34.202110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.202018 4880 generic.go:356] "Generic (PLEG): container finished" podID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerID="7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275" exitCode=0 Oct 31 16:39:34.202110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.202071 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" Oct 31 16:39:34.202365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.202102 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" event={"ID":"ffc1fd80-a78f-4942-b879-07baf07ee37a","Type":"ContainerDied","Data":"7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275"} Oct 31 16:39:34.202365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.202169 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5" event={"ID":"ffc1fd80-a78f-4942-b879-07baf07ee37a","Type":"ContainerDied","Data":"49d9d29f4e94b64a59a2dd5789991c7a115ad9d9b447223380a5737c8fc73930"} Oct 31 16:39:34.202365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.202194 4880 scope.go:122] "RemoveContainer" containerID="7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275" Oct 31 16:39:34.212990 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.212974 4880 scope.go:122] "RemoveContainer" containerID="7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275" Oct 31 16:39:34.213329 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:39:34.213310 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275\": container with ID starting with 7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275 not found: ID does not exist" containerID="7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275" Oct 31 16:39:34.213398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.213339 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275"} err="failed to get container status \"7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275\": rpc error: code = NotFound desc = could not find container \"7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275\": container with ID starting with 7dbcfbf31d490a5a43ede89624c901be32fdc8e19382a5e52782628472fba275 not found: ID does not exist" Oct 31 16:39:34.355926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.355892 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5"] Oct 31 16:39:34.359936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.359910 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-59357a-openshift-default-5b84865d9c-k2mk5"] Oct 31 16:39:34.801871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:34.801837 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:35.091243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:35.091165 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" path="/var/lib/kubelet/pods/ffc1fd80-a78f-4942-b879-07baf07ee37a/volumes" Oct 31 16:39:35.480487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:35.480448 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:36.800806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:36.800770 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f"] Oct 31 16:39:36.846702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:36.846670 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875"] Oct 31 16:39:36.986680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:36.986646 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:39.678730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:39.678678 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:41.385799 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:41.385761 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:41.561680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:41.561640 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:41.733092 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:41.732975 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:41.902404 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:41.902359 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:42.179169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:42.179090 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:42.328961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:42.328927 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:42.486270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:42.486162 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:43.032103 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.032070 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:43.558919 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.558881 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:43.760467 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.760425 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:43.823009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.822930 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq"] Oct 31 16:39:43.823555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823537 4880 cpu_manager.go:414] "containerMap: removing container" podUID="83c249f9-8a9a-46f7-b502-6cf2d21a2508" containerName="maas-api" Oct 31 16:39:43.823616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823559 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c249f9-8a9a-46f7-b502-6cf2d21a2508" containerName="maas-api" Oct 31 16:39:43.823616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823589 4880 cpu_manager.go:414] "containerMap: removing container" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" containerName="test-curl-7204-0147c6" Oct 31 16:39:43.823616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823594 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" containerName="test-curl-7204-0147c6" Oct 31 16:39:43.823616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823604 4880 cpu_manager.go:414] "containerMap: removing container" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" Oct 31 16:39:43.823616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823608 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823623 4880 cpu_manager.go:414] "containerMap: removing container" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823627 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823637 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823642 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823731 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa80a393-96cf-4c65-8d56-84d6ca0292b8" containerName="main" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823741 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3059189-8881-400a-9fa4-4cef8b277a8d" containerName="istio-proxy" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823752 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c249f9-8a9a-46f7-b502-6cf2d21a2508" containerName="maas-api" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823760 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5d8be9-7938-4703-9649-480ef3125f69" containerName="test-curl-7204-0147c6" Oct 31 16:39:43.823783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.823766 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc1fd80-a78f-4942-b879-07baf07ee37a" containerName="istio-proxy" Oct 31 16:39:43.827083 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.827062 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.923214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.923184 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq"] Oct 31 16:39:43.974617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.974579 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.974800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.974744 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b228db-d145-47dd-8609-76d77564749e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.974800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.974784 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.974903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.974888 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwng\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-kube-api-access-dfwng\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.974941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.974925 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.975122 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.975098 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b228db-d145-47dd-8609-76d77564749e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.975173 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.975150 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.975223 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.975170 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:43.975316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:43.975295 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076065 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b228db-d145-47dd-8609-76d77564749e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076107 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076157 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076127 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076193 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076242 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076324 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b228db-d145-47dd-8609-76d77564749e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076358 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.076450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076423 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwng\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-kube-api-access-dfwng\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.077055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076456 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.077055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076510 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.077055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076607 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.077055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076676 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.077055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076921 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b228db-d145-47dd-8609-76d77564749e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.077055 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.076955 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.078760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.078742 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.078972 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.078956 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b228db-d145-47dd-8609-76d77564749e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.130689 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.130661 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.130842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.130826 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwng\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-kube-api-access-dfwng\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m7bsq\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.144699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.144668 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:44.575641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.575602 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq"] Oct 31 16:39:44.582039 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:39:44.582006 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b228db_d145_47dd_8609_76d77564749e.slice/crio-d00b7a6a4abfa8dbd7bc8f3f3a6c1fdfaa99d90610550cd009a6e06a2536db90 WatchSource:0}: Error finding container d00b7a6a4abfa8dbd7bc8f3f3a6c1fdfaa99d90610550cd009a6e06a2536db90: Status 404 returned error can't find the container with id d00b7a6a4abfa8dbd7bc8f3f3a6c1fdfaa99d90610550cd009a6e06a2536db90 Oct 31 16:39:44.586325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.586273 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:39:44.586437 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.586374 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:39:44.586437 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:44.586415 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:39:45.296501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:45.296467 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" event={"ID":"f7b228db-d145-47dd-8609-76d77564749e","Type":"ContainerStarted","Data":"8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5"} Oct 31 16:39:45.296683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:45.296524 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" event={"ID":"f7b228db-d145-47dd-8609-76d77564749e","Type":"ContainerStarted","Data":"d00b7a6a4abfa8dbd7bc8f3f3a6c1fdfaa99d90610550cd009a6e06a2536db90"} Oct 31 16:39:45.407502 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:45.407447 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podStartSLOduration=2.407424204 podStartE2EDuration="2.407424204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:39:45.40500664 +0000 UTC m=+2232.440301464" watchObservedRunningTime="2026-10-31 16:39:45.407424204 +0000 UTC m=+2232.442719008" Oct 31 16:39:46.145380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:46.145331 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:46.147110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:46.147083 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m7bsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 16:39:46.147260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:46.147150 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 16:39:47.047260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:47.047218 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:47.152630 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:47.152588 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m7bsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 16:39:47.153236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:47.152655 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 16:39:47.379755 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:47.379656 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:48.145666 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:48.145626 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m7bsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 16:39:48.145887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:48.145699 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 16:39:48.557586 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:48.557505 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:48.753430 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:48.753387 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:49.145685 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:49.145645 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m7bsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 16:39:49.145904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:49.145725 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 16:39:50.156400 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:50.156361 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m7bsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 16:39:50.156906 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:50.156541 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 16:39:51.145995 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:51.145956 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m7bsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 16:39:51.146196 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:51.146016 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 16:39:51.382629 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:51.382590 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:51.821904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:51.821859 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" containerID="cri-o://e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3" gracePeriod=45 Oct 31 16:39:51.867825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:51.867775 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" containerID="cri-o://d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508" gracePeriod=45 Oct 31 16:39:52.146071 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.146040 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m7bsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 31 16:39:52.146236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.146106 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 31 16:39:52.165828 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.165803 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:39:52.169270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.169249 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.208681 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-dshm\") pod \"05265cb6-63be-4ce4-b005-673ece77fb83\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.208809 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-tmp-dir\") pod \"05265cb6-63be-4ce4-b005-673ece77fb83\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.208848 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05265cb6-63be-4ce4-b005-673ece77fb83-tls-certs\" (UniqueName: \"kubernetes.io/secret/05265cb6-63be-4ce4-b005-673ece77fb83-tls-certs\") pod \"05265cb6-63be-4ce4-b005-673ece77fb83\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.208927 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tmp-dir\") pod \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.208964 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tls-certs\") pod \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.209051 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-home\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-home\") pod \"05265cb6-63be-4ce4-b005-673ece77fb83\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.209098 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-model-cache\") pod \"05265cb6-63be-4ce4-b005-673ece77fb83\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.209155 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-dshm\") pod \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.209228 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-kube-api-access-2gt4c\" (UniqueName: \"kubernetes.io/projected/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-kube-api-access-2gt4c\") pod \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.209270 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-model-cache\") pod \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.209351 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-home\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-home\") pod \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\" (UID: \"d43fd7b4-2fb6-49a4-9902-c1f7b5589784\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.209381 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05265cb6-63be-4ce4-b005-673ece77fb83-kube-api-access-mswqh\" (UniqueName: \"kubernetes.io/projected/05265cb6-63be-4ce4-b005-673ece77fb83-kube-api-access-mswqh\") pod \"05265cb6-63be-4ce4-b005-673ece77fb83\" (UID: \"05265cb6-63be-4ce4-b005-673ece77fb83\") " Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.210889 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-home" pod "05265cb6-63be-4ce4-b005-673ece77fb83" (UID: "05265cb6-63be-4ce4-b005-673ece77fb83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.216882 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-model-cache" pod "d43fd7b4-2fb6-49a4-9902-c1f7b5589784" (UID: "d43fd7b4-2fb6-49a4-9902-c1f7b5589784"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.217326 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tmp-dir" pod "d43fd7b4-2fb6-49a4-9902-c1f7b5589784" (UID: "d43fd7b4-2fb6-49a4-9902-c1f7b5589784"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.217401 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-model-cache" pod "05265cb6-63be-4ce4-b005-673ece77fb83" (UID: "05265cb6-63be-4ce4-b005-673ece77fb83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.217835 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-tmp-dir" pod "05265cb6-63be-4ce4-b005-673ece77fb83" (UID: "05265cb6-63be-4ce4-b005-673ece77fb83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.218206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.218196 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-home" pod "d43fd7b4-2fb6-49a4-9902-c1f7b5589784" (UID: "d43fd7b4-2fb6-49a4-9902-c1f7b5589784"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.327165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.316537 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.327165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.316585 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.327165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.316598 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.327165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.316613 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.327165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.316626 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.327165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.316639 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.354802 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.352721 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-dshm" pod "05265cb6-63be-4ce4-b005-673ece77fb83" (UID: "05265cb6-63be-4ce4-b005-673ece77fb83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.355472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.355432 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-dshm" pod "d43fd7b4-2fb6-49a4-9902-c1f7b5589784" (UID: "d43fd7b4-2fb6-49a4-9902-c1f7b5589784"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:39:52.355472 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.355460 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-kube-api-access-2gt4c" pod "d43fd7b4-2fb6-49a4-9902-c1f7b5589784" (UID: "d43fd7b4-2fb6-49a4-9902-c1f7b5589784"). InnerVolumeSpecName "kube-api-access-2gt4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:52.355657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.355443 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05265cb6-63be-4ce4-b005-673ece77fb83-kube-api-access-mswqh" pod "05265cb6-63be-4ce4-b005-673ece77fb83" (UID: "05265cb6-63be-4ce4-b005-673ece77fb83"). InnerVolumeSpecName "kube-api-access-mswqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:39:52.355657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.355592 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tls-certs" pod "d43fd7b4-2fb6-49a4-9902-c1f7b5589784" (UID: "d43fd7b4-2fb6-49a4-9902-c1f7b5589784"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:39:52.356973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.356947 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05265cb6-63be-4ce4-b005-673ece77fb83-tls-certs" pod "05265cb6-63be-4ce4-b005-673ece77fb83" (UID: "05265cb6-63be-4ce4-b005-673ece77fb83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:39:52.379886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.379790 4880 generic.go:356] "Generic (PLEG): container finished" podID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerID="e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3" exitCode=0 Oct 31 16:39:52.379886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.379849 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" Oct 31 16:39:52.380101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.379883 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" event={"ID":"d43fd7b4-2fb6-49a4-9902-c1f7b5589784","Type":"ContainerDied","Data":"e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3"} Oct 31 16:39:52.380101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.379939 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f" event={"ID":"d43fd7b4-2fb6-49a4-9902-c1f7b5589784","Type":"ContainerDied","Data":"b341b5a491537f057a06d3df29552e51fb376aefcda593532a30acc0acbd27bd"} Oct 31 16:39:52.380101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.379962 4880 scope.go:122] "RemoveContainer" containerID="e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3" Oct 31 16:39:52.382063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.381860 4880 generic.go:356] "Generic (PLEG): container finished" podID="05265cb6-63be-4ce4-b005-673ece77fb83" containerID="d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508" exitCode=0 Oct 31 16:39:52.382063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.381890 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" event={"ID":"05265cb6-63be-4ce4-b005-673ece77fb83","Type":"ContainerDied","Data":"d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508"} Oct 31 16:39:52.382063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.381912 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" event={"ID":"05265cb6-63be-4ce4-b005-673ece77fb83","Type":"ContainerDied","Data":"9e08e258a14d4810ee6225a2e1cc2346aed28e16751678cbebe1057dcc0a56ed"} Oct 31 16:39:52.382063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.381921 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875" Oct 31 16:39:52.396897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.396874 4880 scope.go:122] "RemoveContainer" containerID="e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3" Oct 31 16:39:52.397399 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:39:52.397372 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3\": container with ID starting with e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3 not found: ID does not exist" containerID="e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3" Oct 31 16:39:52.397477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.397435 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3"} err="failed to get container status \"e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3\": rpc error: code = NotFound desc = could not find container \"e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3\": container with ID starting with e53deb27e891e8ee305f40d7f3847eaa8729ea5bebb9c375d601f78b0107abe3 not found: ID does not exist" Oct 31 16:39:52.397477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.397465 4880 scope.go:122] "RemoveContainer" containerID="d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508" Oct 31 16:39:52.408304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.408280 4880 scope.go:122] "RemoveContainer" containerID="d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508" Oct 31 16:39:52.408658 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:39:52.408635 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508\": container with ID starting with d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508 not found: ID does not exist" containerID="d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508" Oct 31 16:39:52.408738 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.408669 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508"} err="failed to get container status \"d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508\": rpc error: code = NotFound desc = could not find container \"d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508\": container with ID starting with d5e54edb716890a8c8cd2bd8a5844ac335bcddb9ed2caa5d1d2bdcf6c8b7f508 not found: ID does not exist" Oct 31 16:39:52.418388 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.418362 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05265cb6-63be-4ce4-b005-673ece77fb83-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.418388 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.418388 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.418522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.418401 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.418522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.418417 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gt4c\" (UniqueName: \"kubernetes.io/projected/d43fd7b4-2fb6-49a4-9902-c1f7b5589784-kube-api-access-2gt4c\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.418522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.418431 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mswqh\" (UniqueName: \"kubernetes.io/projected/05265cb6-63be-4ce4-b005-673ece77fb83-kube-api-access-mswqh\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.418522 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.418443 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05265cb6-63be-4ce4-b005-673ece77fb83-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:39:52.530964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.530902 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875"] Oct 31 16:39:52.540626 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.540592 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/simulator-w1-4fe3a4-kserve-544f5d68cc-wt875"] Oct 31 16:39:52.635031 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.635004 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f"] Oct 31 16:39:52.640839 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.640813 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4fe3a4/premium-w1-4fe3a4-kserve-56cbbc644d-t258f"] Oct 31 16:39:52.975761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:52.975675 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:53.091264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.091233 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" path="/var/lib/kubelet/pods/05265cb6-63be-4ce4-b005-673ece77fb83/volumes" Oct 31 16:39:53.091724 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.091711 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" path="/var/lib/kubelet/pods/d43fd7b4-2fb6-49a4-9902-c1f7b5589784/volumes" Oct 31 16:39:53.255880 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.255787 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:53.393100 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.393049 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:53.394942 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.394915 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:53.397090 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.397046 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:39:53.484415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.484380 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:53.909782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:53.909725 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:39:58.459128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:39:58.459088 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:00.312523 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:00.312486 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:00.491606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:00.491571 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:00.658565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:00.658530 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:03.555879 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:03.555836 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:03.883789 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:03.883739 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:40:08.714948 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:08.714855 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:10.514544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:10.514502 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:12.364861 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:12.364823 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:13.262000 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:13.261967 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:13.461665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:13.461618 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:13.610115 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:13.610017 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:14.529842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:14.529802 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:14.680178 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:14.680126 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:14.841212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:14.841088 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:17.988770 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:17.988735 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:18.353185 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:18.353084 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:20.325933 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:20.325890 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:20.473756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:20.473708 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:23.507810 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:23.507772 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:23.652566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:23.652528 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:24.638529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:24.638490 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:25.295723 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:25.295676 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7"] Oct 31 16:40:25.442841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:25.442804 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:29.098318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:29.098279 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:30.245762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:30.245711 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:32.638896 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:32.638861 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:32.799282 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:32.799245 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:33.567937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.567901 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws"] Oct 31 16:40:33.568129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.568095 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" podUID="3c5ec607-72b9-44e6-ad3f-0dd99140ab00" containerName="maas-api" containerID="cri-o://3a20984cc80b4d14cd77b87f9e72cef73a3fbdbe9fa6dbfc964136f4ee2c41f5" gracePeriod=30 Oct 31 16:40:33.687308 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.687200 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:33.763255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.763221 4880 generic.go:356] "Generic (PLEG): container finished" podID="3c5ec607-72b9-44e6-ad3f-0dd99140ab00" containerID="3a20984cc80b4d14cd77b87f9e72cef73a3fbdbe9fa6dbfc964136f4ee2c41f5" exitCode=0 Oct 31 16:40:33.763452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.763396 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" event={"ID":"3c5ec607-72b9-44e6-ad3f-0dd99140ab00","Type":"ContainerDied","Data":"3a20984cc80b4d14cd77b87f9e72cef73a3fbdbe9fa6dbfc964136f4ee2c41f5"} Oct 31 16:40:33.876180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.874948 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:40:33.884388 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:33.884354 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:40:34.001710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.993922 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-kube-api-access-p8wpp\" (UniqueName: \"kubernetes.io/projected/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-kube-api-access-p8wpp\") pod \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " Oct 31 16:40:34.001710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.993975 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-openshift-service-ca\") pod \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " Oct 31 16:40:34.001710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.994364 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-metrics-certs\") pod \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " Oct 31 16:40:34.001710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.994458 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls\") pod \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\" (UID: \"3c5ec607-72b9-44e6-ad3f-0dd99140ab00\") " Oct 31 16:40:34.001710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:33.996601 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-openshift-service-ca" pod "3c5ec607-72b9-44e6-ad3f-0dd99140ab00" (UID: "3c5ec607-72b9-44e6-ad3f-0dd99140ab00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:40:34.032533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.032396 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-metrics-certs" pod "3c5ec607-72b9-44e6-ad3f-0dd99140ab00" (UID: "3c5ec607-72b9-44e6-ad3f-0dd99140ab00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:40:34.061744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.052406 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-kube-api-access-p8wpp" pod "3c5ec607-72b9-44e6-ad3f-0dd99140ab00" (UID: "3c5ec607-72b9-44e6-ad3f-0dd99140ab00"). InnerVolumeSpecName "kube-api-access-p8wpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:34.061744 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.053003 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls" pod "3c5ec607-72b9-44e6-ad3f-0dd99140ab00" (UID: "3c5ec607-72b9-44e6-ad3f-0dd99140ab00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:40:34.102292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.099302 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8wpp\" (UniqueName: \"kubernetes.io/projected/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-kube-api-access-p8wpp\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:34.102292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.099347 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:34.102292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.099364 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:34.102292 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.099379 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c5ec607-72b9-44e6-ad3f-0dd99140ab00-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:34.770971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.770943 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" Oct 31 16:40:34.771420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.770975 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws" event={"ID":"3c5ec607-72b9-44e6-ad3f-0dd99140ab00","Type":"ContainerDied","Data":"18ceed79f4f4063409f00eb324752bd555be38fd0e0b06ac68ae47f1d62badce"} Oct 31 16:40:34.771420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.771036 4880 scope.go:122] "RemoveContainer" containerID="3a20984cc80b4d14cd77b87f9e72cef73a3fbdbe9fa6dbfc964136f4ee2c41f5" Oct 31 16:40:34.926928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.926891 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws"] Oct 31 16:40:34.970639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:34.970603 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e7f13a-777846cbcd-78kws"] Oct 31 16:40:35.091245 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:35.091153 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5ec607-72b9-44e6-ad3f-0dd99140ab00" path="/var/lib/kubelet/pods/3c5ec607-72b9-44e6-ad3f-0dd99140ab00/volumes" Oct 31 16:40:35.340296 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:35.340262 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:35.552606 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:35.552571 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:35.707237 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:35.707205 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:35.856776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:35.856701 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:36.272750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:36.272716 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:40.317167 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.317092 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" containerID="cri-o://ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8" gracePeriod=45 Oct 31 16:40:40.481275 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.481238 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused" Oct 31 16:40:40.678561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.678530 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:40:40.782264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782225 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-model-cache\") pod \"af2f32e8-885f-4eae-8301-54da9247f464\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " Oct 31 16:40:40.782264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782267 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-tmp-dir\") pod \"af2f32e8-885f-4eae-8301-54da9247f464\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " Oct 31 16:40:40.782529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782293 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2f32e8-885f-4eae-8301-54da9247f464-kube-api-access-q4vkn\" (UniqueName: \"kubernetes.io/projected/af2f32e8-885f-4eae-8301-54da9247f464-kube-api-access-q4vkn\") pod \"af2f32e8-885f-4eae-8301-54da9247f464\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " Oct 31 16:40:40.782529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782337 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-home\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-home\") pod \"af2f32e8-885f-4eae-8301-54da9247f464\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " Oct 31 16:40:40.782529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782441 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-dshm\") pod \"af2f32e8-885f-4eae-8301-54da9247f464\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " Oct 31 16:40:40.782706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782535 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af2f32e8-885f-4eae-8301-54da9247f464-tls-certs\" (UniqueName: \"kubernetes.io/secret/af2f32e8-885f-4eae-8301-54da9247f464-tls-certs\") pod \"af2f32e8-885f-4eae-8301-54da9247f464\" (UID: \"af2f32e8-885f-4eae-8301-54da9247f464\") " Oct 31 16:40:40.782706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782549 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-model-cache" pod "af2f32e8-885f-4eae-8301-54da9247f464" (UID: "af2f32e8-885f-4eae-8301-54da9247f464"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:40.782706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782566 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-tmp-dir" pod "af2f32e8-885f-4eae-8301-54da9247f464" (UID: "af2f32e8-885f-4eae-8301-54da9247f464"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:40.782706 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.782667 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-home" pod "af2f32e8-885f-4eae-8301-54da9247f464" (UID: "af2f32e8-885f-4eae-8301-54da9247f464"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:40.783121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.783094 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:40.783121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.783121 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:40.783415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.783148 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:40.785206 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.785177 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-dshm" pod "af2f32e8-885f-4eae-8301-54da9247f464" (UID: "af2f32e8-885f-4eae-8301-54da9247f464"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:40.785360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.785341 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2f32e8-885f-4eae-8301-54da9247f464-kube-api-access-q4vkn" pod "af2f32e8-885f-4eae-8301-54da9247f464" (UID: "af2f32e8-885f-4eae-8301-54da9247f464"). InnerVolumeSpecName "kube-api-access-q4vkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:40.786592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.786568 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af2f32e8-885f-4eae-8301-54da9247f464-tls-certs" pod "af2f32e8-885f-4eae-8301-54da9247f464" (UID: "af2f32e8-885f-4eae-8301-54da9247f464"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:40:40.822073 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.822039 4880 generic.go:356] "Generic (PLEG): container finished" podID="af2f32e8-885f-4eae-8301-54da9247f464" containerID="ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8" exitCode=0 Oct 31 16:40:40.822324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.822115 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" Oct 31 16:40:40.822324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.822246 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" event={"ID":"af2f32e8-885f-4eae-8301-54da9247f464","Type":"ContainerDied","Data":"ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8"} Oct 31 16:40:40.822324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.822276 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7" event={"ID":"af2f32e8-885f-4eae-8301-54da9247f464","Type":"ContainerDied","Data":"dd8d3fb680331caeaeeff780af1f19cc7db5219f5845f08a9df8d88f968a5b22"} Oct 31 16:40:40.822324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.822293 4880 scope.go:122] "RemoveContainer" containerID="ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8" Oct 31 16:40:40.841215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.841095 4880 scope.go:122] "RemoveContainer" containerID="ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8" Oct 31 16:40:40.841594 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:40.841513 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8\": container with ID starting with ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8 not found: ID does not exist" containerID="ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8" Oct 31 16:40:40.841594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.841556 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8"} err="failed to get container status \"ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8\": rpc error: code = NotFound desc = could not find container \"ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8\": container with ID starting with ee8f13322c953a49682a4985fb105733f1df00e64aca146e004527403b2937a8 not found: ID does not exist" Oct 31 16:40:40.885031 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.884979 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af2f32e8-885f-4eae-8301-54da9247f464-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:40.885031 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.885031 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af2f32e8-885f-4eae-8301-54da9247f464-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:40.885301 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.885051 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vkn\" (UniqueName: \"kubernetes.io/projected/af2f32e8-885f-4eae-8301-54da9247f464-kube-api-access-q4vkn\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:40.984297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.984100 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7"] Oct 31 16:40:40.992491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:40.992461 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-59357a/distinct-w4-59357a-kserve-5bb9cc867-bv8x7"] Oct 31 16:40:41.090309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:41.090275 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2f32e8-885f-4eae-8301-54da9247f464" path="/var/lib/kubelet/pods/af2f32e8-885f-4eae-8301-54da9247f464/volumes" Oct 31 16:40:41.098487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:41.098459 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:41.724980 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:41.724937 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:43.252227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:43.251825 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:43.418873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:43.418829 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:43.567978 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:43.567881 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv"] Oct 31 16:40:43.574594 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:43.574550 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" containerID="cri-o://3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1" gracePeriod=30 Oct 31 16:40:43.586235 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:43.582130 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:43.850047 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:43.849890 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:44.013361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.013325 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:44.627063 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.627023 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:44.795358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.795315 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:44.912036 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.912005 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65"] Oct 31 16:40:44.912244 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.912216 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" containerID="cri-o://9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d" gracePeriod=30 Oct 31 16:40:44.930715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.930682 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv"] Oct 31 16:40:44.930904 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.930881 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" containerID="cri-o://a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0" gracePeriod=30 Oct 31 16:40:44.950397 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.950360 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:44.953325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.953300 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q"] Oct 31 16:40:44.953528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:44.953507 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" containerID="cri-o://b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a" gracePeriod=30 Oct 31 16:40:45.923988 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:45.923953 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:46.555714 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:46.555670 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:47.396592 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:47.396555 4880 patch_prober.go:29] interesting pod/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:40:47.396984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:47.396613 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:40:48.847574 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.847550 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:40:48.927710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.927680 4880 generic.go:356] "Generic (PLEG): container finished" podID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerID="3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1" exitCode=0 Oct 31 16:40:48.927907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.927729 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" Oct 31 16:40:48.927907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.927759 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" event={"ID":"a28e7e01-7194-4569-90ef-311b5d3b5fa2","Type":"ContainerDied","Data":"3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1"} Oct 31 16:40:48.927907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.927812 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv" event={"ID":"a28e7e01-7194-4569-90ef-311b5d3b5fa2","Type":"ContainerDied","Data":"eb86d27d0afe6f6b6c8db3f05eee4ec6615bab2cc7d266d1323f91567a1d22fc"} Oct 31 16:40:48.927907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.927834 4880 scope.go:122] "RemoveContainer" containerID="3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1" Oct 31 16:40:48.936269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936245 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-certs\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936278 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-data\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936307 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-podinfo\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936388 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-token\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-token\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936437 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-envoy\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936502 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-socket\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936526 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-data" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:48.936551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936539 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-certs" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:48.936703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936534 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-credential-socket\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936596 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istiod-ca-cert\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936703 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936652 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-kube-api-access-zwjq8\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-kube-api-access-zwjq8\") pod \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\" (UID: \"a28e7e01-7194-4569-90ef-311b5d3b5fa2\") " Oct 31 16:40:48.936865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936773 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-credential-socket" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:48.936865 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.936786 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-socket" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:48.937567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.937232 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istiod-ca-cert" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:40:48.937567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.937475 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:48.937567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.937499 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:48.937567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.937515 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:48.937567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.937529 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:48.937567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.937543 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:48.938579 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.938481 4880 scope.go:122] "RemoveContainer" containerID="3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1" Oct 31 16:40:48.938708 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.938682 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-token" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:48.938782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.938766 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-podinfo" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:40:48.938839 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:48.938798 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1\": container with ID starting with 3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1 not found: ID does not exist" containerID="3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1" Oct 31 16:40:48.938881 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.938838 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1"} err="failed to get container status \"3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1\": rpc error: code = NotFound desc = could not find container \"3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1\": container with ID starting with 3c5139d564d545970758744afb2e8113b9a764fdf76ce5ae4ff7666ccc7cf6c1 not found: ID does not exist" Oct 31 16:40:48.939095 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.939073 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-kube-api-access-zwjq8" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "kube-api-access-zwjq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:48.939207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:48.939188 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-envoy" pod "a28e7e01-7194-4569-90ef-311b5d3b5fa2" (UID: "a28e7e01-7194-4569-90ef-311b5d3b5fa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:49.039261 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:49.039222 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjq8\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-kube-api-access-zwjq8\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:49.039261 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:49.039255 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:49.039261 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:49.039267 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:49.039489 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:49.039276 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28e7e01-7194-4569-90ef-311b5d3b5fa2-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:49.336320 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:49.336228 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv"] Oct 31 16:40:49.339660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:49.339634 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7f13a-openshift-default-d9477795f-xsxhv"] Oct 31 16:40:50.243199 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.238252 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:40:50.265330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.265304 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:40:50.284199 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.284179 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:40:50.355633 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.355600 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-credential-socket\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.355818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.355643 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-podinfo\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.355818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.355671 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-certs\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.355818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.355704 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-istio-token\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-istio-token\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.355818 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.355746 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-envoy\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.356091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.355941 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-credential-socket" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.356091 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356048 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-certs" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.356300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356282 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-data\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.356383 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356318 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-socket\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.356383 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356372 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f02272f-ae28-4942-80c0-0d08040a767e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f02272f-ae28-4942-80c0-0d08040a767e-istio-podinfo\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.356491 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356414 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-kube-api-access-bqpn2\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-kube-api-access-bqpn2\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.356546 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356484 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-credential-socket\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.356546 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356532 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f02272f-ae28-4942-80c0-0d08040a767e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f02272f-ae28-4942-80c0-0d08040a767e-istiod-ca-cert\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.356705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356577 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d2bc6c-026d-4006-878c-4697689f8a32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d2bc6c-026d-4006-878c-4697689f8a32-istiod-ca-cert\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.356705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356624 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-token\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-token\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.356705 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356675 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-envoy\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.356857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356704 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-kube-api-access-r46wj\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-kube-api-access-r46wj\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.356857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356761 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-socket\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.356857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356833 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-data\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.357038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356880 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-data\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.357038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356936 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17d2bc6c-026d-4006-878c-4697689f8a32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d2bc6c-026d-4006-878c-4697689f8a32-istio-podinfo\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.357038 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.356977 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-credential-socket\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.357227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.357036 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-socket\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.357227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.357066 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-envoy\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.357227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.357104 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179e2ff2-6196-4d03-bc6d-74397af48f56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179e2ff2-6196-4d03-bc6d-74397af48f56-istiod-ca-cert\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.357227 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.357182 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-certs\") pod \"8f02272f-ae28-4942-80c0-0d08040a767e\" (UID: \"8f02272f-ae28-4942-80c0-0d08040a767e\") " Oct 31 16:40:50.357430 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.357224 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-kube-api-access-qf2pd\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-kube-api-access-qf2pd\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.357971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.357908 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-credential-socket" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.358380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.358353 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-credential-socket" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.358466 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.358387 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-socket" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.358609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.358590 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-data" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.358646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.358593 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-socket" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.358789 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.358769 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-data" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.359093 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359069 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-data" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.359380 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.357260 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-certs\") pod \"179e2ff2-6196-4d03-bc6d-74397af48f56\" (UID: \"179e2ff2-6196-4d03-bc6d-74397af48f56\") " Oct 31 16:40:50.359479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359406 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-istio-token\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-istio-token\") pod \"17d2bc6c-026d-4006-878c-4697689f8a32\" (UID: \"17d2bc6c-026d-4006-878c-4697689f8a32\") " Oct 31 16:40:50.359863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359843 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359866 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359879 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359892 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359905 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359918 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359931 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359944 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.359969 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.359958 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.360907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.360623 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-certs" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.360907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.360877 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179e2ff2-6196-4d03-bc6d-74397af48f56-istiod-ca-cert" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:40:50.361104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.360942 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-certs" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.361104 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361080 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f02272f-ae28-4942-80c0-0d08040a767e-istiod-ca-cert" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:40:50.361499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361466 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-socket" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.361679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361657 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-envoy" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.361755 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361733 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f02272f-ae28-4942-80c0-0d08040a767e-istio-podinfo" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:40:50.361811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361773 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-kube-api-access-r46wj" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "kube-api-access-r46wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:50.361811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361800 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-istio-token" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:50.361935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361816 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d2bc6c-026d-4006-878c-4697689f8a32-istiod-ca-cert" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:40:50.361935 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.361913 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17d2bc6c-026d-4006-878c-4697689f8a32-istio-podinfo" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:40:50.362250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.362227 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-kube-api-access-bqpn2" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "kube-api-access-bqpn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:50.362560 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.362536 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-podinfo" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:40:50.363037 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.363015 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-kube-api-access-qf2pd" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "kube-api-access-qf2pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:50.363749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.363713 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-token" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:50.363975 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.363946 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-istio-token" pod "17d2bc6c-026d-4006-878c-4697689f8a32" (UID: "17d2bc6c-026d-4006-878c-4697689f8a32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:50.364809 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.364781 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-envoy" pod "8f02272f-ae28-4942-80c0-0d08040a767e" (UID: "8f02272f-ae28-4942-80c0-0d08040a767e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.364891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.364828 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-envoy" pod "179e2ff2-6196-4d03-bc6d-74397af48f56" (UID: "179e2ff2-6196-4d03-bc6d-74397af48f56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:40:50.462018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.461931 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.461971 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf2pd\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-kube-api-access-qf2pd\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.461987 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462002 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462016 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462028 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462044 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462059 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f02272f-ae28-4942-80c0-0d08040a767e-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462073 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqpn2\" (UniqueName: \"kubernetes.io/projected/17d2bc6c-026d-4006-878c-4697689f8a32-kube-api-access-bqpn2\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462088 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f02272f-ae28-4942-80c0-0d08040a767e-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462102 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17d2bc6c-026d-4006-878c-4697689f8a32-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462115 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462128 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/179e2ff2-6196-4d03-bc6d-74397af48f56-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462179 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46wj\" (UniqueName: \"kubernetes.io/projected/8f02272f-ae28-4942-80c0-0d08040a767e-kube-api-access-r46wj\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462195 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17d2bc6c-026d-4006-878c-4697689f8a32-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462221 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17d2bc6c-026d-4006-878c-4697689f8a32-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462236 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f02272f-ae28-4942-80c0-0d08040a767e-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.462420 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.462249 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/179e2ff2-6196-4d03-bc6d-74397af48f56-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:50.944728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.944693 4880 generic.go:356] "Generic (PLEG): container finished" podID="17d2bc6c-026d-4006-878c-4697689f8a32" containerID="9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d" exitCode=0 Oct 31 16:40:50.944890 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.944757 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" Oct 31 16:40:50.944890 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.944773 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" event={"ID":"17d2bc6c-026d-4006-878c-4697689f8a32","Type":"ContainerDied","Data":"9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d"} Oct 31 16:40:50.944890 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.944855 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65" event={"ID":"17d2bc6c-026d-4006-878c-4697689f8a32","Type":"ContainerDied","Data":"25a49c1512c3c2a9da68de89dbf9e692d33b180961cff7edb00a423734238f2f"} Oct 31 16:40:50.945057 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.944891 4880 scope.go:122] "RemoveContainer" containerID="9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d" Oct 31 16:40:50.946479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.946455 4880 generic.go:356] "Generic (PLEG): container finished" podID="8f02272f-ae28-4942-80c0-0d08040a767e" containerID="b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a" exitCode=0 Oct 31 16:40:50.946581 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.946508 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" Oct 31 16:40:50.946581 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.946561 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" event={"ID":"8f02272f-ae28-4942-80c0-0d08040a767e","Type":"ContainerDied","Data":"b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a"} Oct 31 16:40:50.946684 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.946639 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q" event={"ID":"8f02272f-ae28-4942-80c0-0d08040a767e","Type":"ContainerDied","Data":"5121830ea0ec2c8edc1e3266dcec8f03a075acaa2abe5f05e150f43a53766d25"} Oct 31 16:40:50.948325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.948303 4880 generic.go:356] "Generic (PLEG): container finished" podID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerID="a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0" exitCode=0 Oct 31 16:40:50.948419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.948341 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" event={"ID":"179e2ff2-6196-4d03-bc6d-74397af48f56","Type":"ContainerDied","Data":"a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0"} Oct 31 16:40:50.948419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.948374 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" Oct 31 16:40:50.948419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.948404 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv" event={"ID":"179e2ff2-6196-4d03-bc6d-74397af48f56","Type":"ContainerDied","Data":"edde7aa4e3a0d202487a47beb5f33fff3684480f113c775f49b7a9a270cdf4a9"} Oct 31 16:40:50.958827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.958808 4880 scope.go:122] "RemoveContainer" containerID="9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d" Oct 31 16:40:50.959130 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:50.959105 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d\": container with ID starting with 9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d not found: ID does not exist" containerID="9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d" Oct 31 16:40:50.959217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.959156 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d"} err="failed to get container status \"9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d\": rpc error: code = NotFound desc = could not find container \"9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d\": container with ID starting with 9a9306ec2708f1e1a6939c7f1727c5a489e5bc96739aab1e7db5fc534e14645d not found: ID does not exist" Oct 31 16:40:50.959217 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.959180 4880 scope.go:122] "RemoveContainer" containerID="b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a" Oct 31 16:40:50.968603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.968585 4880 scope.go:122] "RemoveContainer" containerID="b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a" Oct 31 16:40:50.968836 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:50.968818 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a\": container with ID starting with b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a not found: ID does not exist" containerID="b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a" Oct 31 16:40:50.968894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.968843 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a"} err="failed to get container status \"b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a\": rpc error: code = NotFound desc = could not find container \"b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a\": container with ID starting with b77a02a21b938d75eead7dd9a047b479e3332b661f90392e4658f5b7fb52a68a not found: ID does not exist" Oct 31 16:40:50.968894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.968863 4880 scope.go:122] "RemoveContainer" containerID="a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0" Oct 31 16:40:50.977888 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.977861 4880 scope.go:122] "RemoveContainer" containerID="a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0" Oct 31 16:40:50.978165 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:50.978121 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0\": container with ID starting with a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0 not found: ID does not exist" containerID="a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0" Oct 31 16:40:50.978253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:50.978161 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0"} err="failed to get container status \"a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0\": rpc error: code = NotFound desc = could not find container \"a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0\": container with ID starting with a1a9716f28ab18602f096bd10731a622c58b96e56c2b8519eea0f4acbc3a5da0 not found: ID does not exist" Oct 31 16:40:51.090883 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.090855 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" path="/var/lib/kubelet/pods/a28e7e01-7194-4569-90ef-311b5d3b5fa2/volumes" Oct 31 16:40:51.098611 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.098585 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q"] Oct 31 16:40:51.144459 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.144430 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-xfb7q"] Oct 31 16:40:51.201971 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.201888 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65"] Oct 31 16:40:51.206030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.206007 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-5bq65"] Oct 31 16:40:51.307303 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.307267 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv"] Oct 31 16:40:51.310555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.310529 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a1b30a15-openshift-default-55cc89b84f-smckv"] Oct 31 16:40:51.478211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:51.478090 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:53.091099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:53.091064 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" path="/var/lib/kubelet/pods/179e2ff2-6196-4d03-bc6d-74397af48f56/volumes" Oct 31 16:40:53.091590 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:53.091575 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" path="/var/lib/kubelet/pods/17d2bc6c-026d-4006-878c-4697689f8a32/volumes" Oct 31 16:40:53.092017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:53.092004 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" path="/var/lib/kubelet/pods/8f02272f-ae28-4942-80c0-0d08040a767e/volumes" Oct 31 16:40:55.443941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.443910 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx"] Oct 31 16:40:55.444366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.444095 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" podUID="b79633c9-3f3e-4d00-9e7f-445219ca919d" containerName="maas-api" containerID="cri-o://adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5" gracePeriod=30 Oct 31 16:40:55.567116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.567085 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:55.755331 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.755301 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:40:55.930808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.930772 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls\") pod \"b79633c9-3f3e-4d00-9e7f-445219ca919d\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " Oct 31 16:40:55.930976 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.930929 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79633c9-3f3e-4d00-9e7f-445219ca919d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79633c9-3f3e-4d00-9e7f-445219ca919d-openshift-service-ca\") pod \"b79633c9-3f3e-4d00-9e7f-445219ca919d\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " Oct 31 16:40:55.930976 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.930952 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79633c9-3f3e-4d00-9e7f-445219ca919d-kube-api-access-7lrvx\" (UniqueName: \"kubernetes.io/projected/b79633c9-3f3e-4d00-9e7f-445219ca919d-kube-api-access-7lrvx\") pod \"b79633c9-3f3e-4d00-9e7f-445219ca919d\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " Oct 31 16:40:55.931053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.931033 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-metrics-certs\") pod \"b79633c9-3f3e-4d00-9e7f-445219ca919d\" (UID: \"b79633c9-3f3e-4d00-9e7f-445219ca919d\") " Oct 31 16:40:55.931326 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.931298 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79633c9-3f3e-4d00-9e7f-445219ca919d-openshift-service-ca" pod "b79633c9-3f3e-4d00-9e7f-445219ca919d" (UID: "b79633c9-3f3e-4d00-9e7f-445219ca919d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:40:55.931536 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.931515 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79633c9-3f3e-4d00-9e7f-445219ca919d-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:55.932945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.932921 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls" pod "b79633c9-3f3e-4d00-9e7f-445219ca919d" (UID: "b79633c9-3f3e-4d00-9e7f-445219ca919d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:40:55.933148 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.933113 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-metrics-certs" pod "b79633c9-3f3e-4d00-9e7f-445219ca919d" (UID: "b79633c9-3f3e-4d00-9e7f-445219ca919d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:40:55.933253 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.933233 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79633c9-3f3e-4d00-9e7f-445219ca919d-kube-api-access-7lrvx" pod "b79633c9-3f3e-4d00-9e7f-445219ca919d" (UID: "b79633c9-3f3e-4d00-9e7f-445219ca919d"). InnerVolumeSpecName "kube-api-access-7lrvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:40:55.998650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.998554 4880 generic.go:356] "Generic (PLEG): container finished" podID="b79633c9-3f3e-4d00-9e7f-445219ca919d" containerID="adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5" exitCode=0 Oct 31 16:40:55.998650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.998598 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" Oct 31 16:40:55.998650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.998622 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" event={"ID":"b79633c9-3f3e-4d00-9e7f-445219ca919d","Type":"ContainerDied","Data":"adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5"} Oct 31 16:40:55.998946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.998666 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx" event={"ID":"b79633c9-3f3e-4d00-9e7f-445219ca919d","Type":"ContainerDied","Data":"5e453fb790d5102e5cdb3720c9ab25ab076279a31948beae4c60acf6a721cbe1"} Oct 31 16:40:55.998946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:55.998690 4880 scope.go:122] "RemoveContainer" containerID="adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5" Oct 31 16:40:56.010930 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:56.010910 4880 scope.go:122] "RemoveContainer" containerID="adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5" Oct 31 16:40:56.011260 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:40:56.011233 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5\": container with ID starting with adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5 not found: ID does not exist" containerID="adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5" Oct 31 16:40:56.011363 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:56.011266 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5"} err="failed to get container status \"adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5\": rpc error: code = NotFound desc = could not find container \"adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5\": container with ID starting with adf70f893e9b33a4f3e9603e01b4a8e12a1ae0b43131ab39d8c95c4b41143ae5 not found: ID does not exist" Oct 31 16:40:56.032259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:56.032234 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrvx\" (UniqueName: \"kubernetes.io/projected/b79633c9-3f3e-4d00-9e7f-445219ca919d-kube-api-access-7lrvx\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:56.032259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:56.032256 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:56.032413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:56.032268 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79633c9-3f3e-4d00-9e7f-445219ca919d-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:40:56.169086 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:56.169048 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx"] Oct 31 16:40:56.172501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:56.172475 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d04acd-78486b4f65-v4vrx"] Oct 31 16:40:57.090738 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:57.090707 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79633c9-3f3e-4d00-9e7f-445219ca919d" path="/var/lib/kubelet/pods/b79633c9-3f3e-4d00-9e7f-445219ca919d/volumes" Oct 31 16:40:57.132663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:57.132628 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:57.412792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:57.412755 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:57.563639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:57.563598 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:58.046528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:58.046485 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:40:59.399232 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.399196 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5"] Oct 31 16:40:59.399946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.399925 4880 cpu_manager.go:414] "containerMap: removing container" podUID="3c5ec607-72b9-44e6-ad3f-0dd99140ab00" containerName="maas-api" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.399947 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5ec607-72b9-44e6-ad3f-0dd99140ab00" containerName="maas-api" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.399973 4880 cpu_manager.go:414] "containerMap: removing container" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.399982 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400001 4880 cpu_manager.go:414] "containerMap: removing container" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400008 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400030 4880 cpu_manager.go:414] "containerMap: removing container" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400037 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" Oct 31 16:40:59.400053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400052 4880 cpu_manager.go:414] "containerMap: removing container" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400058 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400073 4880 cpu_manager.go:414] "containerMap: removing container" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400081 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400093 4880 cpu_manager.go:414] "containerMap: removing container" podUID="b79633c9-3f3e-4d00-9e7f-445219ca919d" containerName="maas-api" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400100 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79633c9-3f3e-4d00-9e7f-445219ca919d" containerName="maas-api" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400115 4880 cpu_manager.go:414] "containerMap: removing container" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400123 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400155 4880 cpu_manager.go:414] "containerMap: removing container" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400163 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400316 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5ec607-72b9-44e6-ad3f-0dd99140ab00" containerName="maas-api" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400335 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d2bc6c-026d-4006-878c-4697689f8a32" containerName="istio-proxy" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400348 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2f32e8-885f-4eae-8301-54da9247f464" containerName="main" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400360 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="179e2ff2-6196-4d03-bc6d-74397af48f56" containerName="istio-proxy" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400370 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="05265cb6-63be-4ce4-b005-673ece77fb83" containerName="main" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400380 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28e7e01-7194-4569-90ef-311b5d3b5fa2" containerName="istio-proxy" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400388 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79633c9-3f3e-4d00-9e7f-445219ca919d" containerName="maas-api" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400398 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43fd7b4-2fb6-49a4-9902-c1f7b5589784" containerName="main" Oct 31 16:40:59.400498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.400408 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f02272f-ae28-4942-80c0-0d08040a767e" containerName="istio-proxy" Oct 31 16:40:59.404116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.404095 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.502188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.502148 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5"] Oct 31 16:40:59.574959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.574915 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.574959 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.574958 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.575221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.574985 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.575221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.575123 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.575221 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.575155 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9v7\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-kube-api-access-cs9v7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.575341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.575260 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.575341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.575285 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.575341 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.575316 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.575475 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.575427 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676300 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676343 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676361 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676414 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676433 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9v7\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-kube-api-access-cs9v7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676487 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676513 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676544 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.676702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676586 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.677014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676755 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.677014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676783 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.677014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676912 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.677014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.676958 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.677283 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.677264 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.678798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.678781 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.678905 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.678889 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.729267 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.729235 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:40:59.729417 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:40:59.729307 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9v7\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-kube-api-access-cs9v7\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ldrj5\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:41:00.018181 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:00.018081 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:41:00.450983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:00.450942 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5"] Oct 31 16:41:00.456332 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:41:00.456299 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77566d1_f156_4cdd_b1ca_fe1e8f6c543b.slice/crio-a4c70b63b64a6060b769ce077061df22758b86a03418ea6542375c8e8ec11593 WatchSource:0}: Error finding container a4c70b63b64a6060b769ce077061df22758b86a03418ea6542375c8e8ec11593: Status 404 returned error can't find the container with id a4c70b63b64a6060b769ce077061df22758b86a03418ea6542375c8e8ec11593 Oct 31 16:41:00.458893 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:00.458856 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:41:00.458986 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:00.458930 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:41:00.458986 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:00.458960 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:41:01.051463 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:01.051416 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" event={"ID":"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b","Type":"ContainerStarted","Data":"36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80"} Oct 31 16:41:01.051642 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:01.051488 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" event={"ID":"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b","Type":"ContainerStarted","Data":"a4c70b63b64a6060b769ce077061df22758b86a03418ea6542375c8e8ec11593"} Oct 31 16:41:01.161288 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:01.161235 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podStartSLOduration=2.161221218 podStartE2EDuration="2.161221218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:41:01.1586231 +0000 UTC m=+2308.193917923" watchObservedRunningTime="2026-10-31 16:41:01.161221218 +0000 UTC m=+2308.196516022" Oct 31 16:41:01.491444 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:01.491400 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:01.646451 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:01.646258 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:02.018989 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:02.018957 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:41:02.021411 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:02.021382 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ldrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 16:41:02.021656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:02.021630 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 16:41:02.489973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:02.489923 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:02.817625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:02.817531 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:03.019270 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:03.019230 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ldrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 16:41:03.019477 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:03.019287 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 16:41:03.682113 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:03.682080 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:03.881465 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:41:03.881430 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:41:03.953797 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:03.953701 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:04.018659 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:04.018626 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ldrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 16:41:04.018820 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:04.018680 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 16:41:04.112097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:04.112063 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:05.019008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:05.018973 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ldrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 16:41:05.019439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:05.019026 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 16:41:06.018718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:06.018679 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ldrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 16:41:06.018894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:06.018735 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 16:41:07.018605 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:07.018569 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ldrj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 31 16:41:07.019070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:07.018627 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 31 16:41:07.759956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:07.759922 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:08.765005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:08.764959 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:41:08.765571 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:08.765553 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:41:08.767158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:08.767123 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:41:09.012175 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:09.012123 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:12.440359 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:12.440321 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:14.141515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:14.141477 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:14.391699 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:14.391601 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:14.550096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:14.550060 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:15.059159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:15.058586 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:16.705722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:16.705688 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:16.905711 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:16.905671 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:19.767746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:19.767709 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:21.132445 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:21.132410 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:21.337515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:21.337480 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:21.454154 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:21.454109 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht"] Oct 31 16:41:21.454419 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:21.454392 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" containerID="cri-o://21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f" gracePeriod=30 Oct 31 16:41:21.510369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:21.510338 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:24.141696 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:24.141660 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:25.729649 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:25.729620 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:25.968369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:25.968329 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:26.124884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.124798 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:26.610025 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.609989 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:26.733567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.733513 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:41:26.855343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855314 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-podinfo\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855365 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-token\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-token\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855413 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istiod-ca-cert\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855465 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-envoy\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855489 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-credential-socket\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855511 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-data\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855534 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-socket\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855644 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-kube-api-access-r685z\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-kube-api-access-r685z\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855716 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-certs\") pod \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\" (UID: \"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3\") " Oct 31 16:41:26.855854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855803 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-data" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:26.855854 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855827 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-credential-socket" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:26.856062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.855979 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-socket" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:26.856062 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.856004 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istiod-ca-cert" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:41:26.856222 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.856189 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-certs" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:26.856325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.856310 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.856365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.856331 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.856365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.856346 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.856365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.856359 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.856474 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.856373 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.857712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.857685 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-token" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:41:26.857946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.857917 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-podinfo" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:41:26.858049 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.857978 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-envoy" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:26.858106 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.858078 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-kube-api-access-r685z" pod "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" (UID: "a7ca0f4d-299b-4495-a6aa-73a6e3aedae3"). InnerVolumeSpecName "kube-api-access-r685z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:41:26.957588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.957555 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.957588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.957583 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r685z\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-kube-api-access-r685z\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.957790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.957599 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:26.957790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:26.957614 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:27.253617 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.253579 4880 generic.go:356] "Generic (PLEG): container finished" podID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerID="21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f" exitCode=0 Oct 31 16:41:27.253787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.253623 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" Oct 31 16:41:27.253787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.253669 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" event={"ID":"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3","Type":"ContainerDied","Data":"21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f"} Oct 31 16:41:27.253787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.253722 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht" event={"ID":"a7ca0f4d-299b-4495-a6aa-73a6e3aedae3","Type":"ContainerDied","Data":"f0b835f4b4d2d52dd2361619194e623983cb73a57542c6cea317b6a6ae2f208e"} Oct 31 16:41:27.253787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.253750 4880 scope.go:122] "RemoveContainer" containerID="21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f" Oct 31 16:41:27.264350 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.264334 4880 scope.go:122] "RemoveContainer" containerID="21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f" Oct 31 16:41:27.264597 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:41:27.264576 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f\": container with ID starting with 21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f not found: ID does not exist" containerID="21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f" Oct 31 16:41:27.264658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.264604 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f"} err="failed to get container status \"21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f\": rpc error: code = NotFound desc = could not find container \"21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f\": container with ID starting with 21f3ee6cfc83c2a5bdb5897439e26de8508032550872a2571699bdc45df9908f not found: ID does not exist" Oct 31 16:41:27.407702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.407675 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht"] Oct 31 16:41:27.453618 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:27.453593 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4fe3a4-openshift-default-5d7894fd77-fhqht"] Oct 31 16:41:29.090442 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:29.090412 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" path="/var/lib/kubelet/pods/a7ca0f4d-299b-4495-a6aa-73a6e3aedae3/volumes" Oct 31 16:41:29.716176 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:29.716111 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:31.150925 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:31.150886 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:32.739717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:32.739678 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:32.938415 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:32.938384 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:33.096450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:33.095831 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:33.403756 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:33.403719 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:33.562048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:33.562003 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:33.882205 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:41:33.882158 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:41:35.990700 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:35.990663 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:36.158601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:36.158565 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:36.319465 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:36.319382 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:37.455726 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:37.455691 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:40.789679 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:40.789645 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:42.519730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:42.519694 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:45.312443 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:45.312406 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:46.989214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:46.989180 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:47.457632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:47.457599 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f"] Oct 31 16:41:49.021392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:49.021356 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:52.607363 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:52.607324 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:53.859023 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:53.858968 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:54.047285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.047248 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:54.141588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.139002 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w"] Oct 31 16:41:54.141588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.139262 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" containerID="cri-o://cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd" gracePeriod=30 Oct 31 16:41:54.162162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.153338 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff"] Oct 31 16:41:54.162162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.153613 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" containerID="cri-o://955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5" gracePeriod=30 Oct 31 16:41:54.212258 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.212147 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:54.637188 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.637129 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:54.821760 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.821722 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:54.970623 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:54.970539 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:56.172914 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:56.172869 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:41:56.173354 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:56.172953 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:41:56.527008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:56.526932 4880 patch_prober.go:29] interesting pod/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:41:56.527008 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:56.526994 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:41:57.665441 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:57.665410 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:59.057211 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.057174 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:59.206680 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.206653 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:59.373174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.366101 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:41:59.449609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.449586 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:41:59.453739 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.453720 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07d7718-5378-4a24-baab-96fbaf7f8e80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d7718-5378-4a24-baab-96fbaf7f8e80-istiod-ca-cert\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.453850 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.453838 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istiod-ca-cert\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.453891 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.453868 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-kube-api-access-z97g5\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-kube-api-access-z97g5\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454005 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.453984 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-podinfo\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454027 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-token\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-token\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454047 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07d7718-5378-4a24-baab-96fbaf7f8e80-istiod-ca-cert" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:41:59.454094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454059 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-data\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454182 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454116 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-data\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454185 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istiod-ca-cert" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:41:59.454229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454190 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-envoy\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454367 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454337 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-data" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.454440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454421 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-token\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-token\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454502 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454470 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-certs\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454499 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-credential-socket\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454504 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-data" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.454563 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454549 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-kube-api-access-9jx2x\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-kube-api-access-9jx2x\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454624 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-socket\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454657 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-envoy\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454693 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-podinfo\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454719 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454708 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-credential-socket" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.454937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454730 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-certs\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454779 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-socket\") pod \"f07d7718-5378-4a24-baab-96fbaf7f8e80\" (UID: \"f07d7718-5378-4a24-baab-96fbaf7f8e80\") " Oct 31 16:41:59.454937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454865 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-credential-socket\") pod \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\" (UID: \"4ba66155-1107-4b7a-9a97-5f0ef951e1cb\") " Oct 31 16:41:59.454937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.454889 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-socket" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.455338 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455295 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-certs" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455484 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d7718-5378-4a24-baab-96fbaf7f8e80-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455506 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455521 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455535 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455549 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455547 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-certs" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455565 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455582 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455679 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-socket" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.455832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.455738 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-credential-socket" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.457075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.456640 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-podinfo" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:41:59.457075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.456700 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-envoy" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.457210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.457073 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-kube-api-access-z97g5" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "kube-api-access-z97g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:41:59.457210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.457097 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-token" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:41:59.457210 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.457195 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-podinfo" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:41:59.457461 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.457435 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-token" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:41:59.457730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.457710 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-envoy" pod "f07d7718-5378-4a24-baab-96fbaf7f8e80" (UID: "f07d7718-5378-4a24-baab-96fbaf7f8e80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:41:59.457827 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.457812 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-kube-api-access-9jx2x" pod "4ba66155-1107-4b7a-9a97-5f0ef951e1cb" (UID: "4ba66155-1107-4b7a-9a97-5f0ef951e1cb"). InnerVolumeSpecName "kube-api-access-9jx2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:41:59.496429 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.496403 4880 generic.go:356] "Generic (PLEG): container finished" podID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerID="cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd" exitCode=0 Oct 31 16:41:59.496537 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.496452 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" Oct 31 16:41:59.496654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.496628 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" event={"ID":"f07d7718-5378-4a24-baab-96fbaf7f8e80","Type":"ContainerDied","Data":"cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd"} Oct 31 16:41:59.496710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.496688 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w" event={"ID":"f07d7718-5378-4a24-baab-96fbaf7f8e80","Type":"ContainerDied","Data":"555a5d65ec89cbd6670181ba1e5f7f3692edc918b90af0f18df796f6b53cb300"} Oct 31 16:41:59.496765 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.496716 4880 scope.go:122] "RemoveContainer" containerID="cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd" Oct 31 16:41:59.498323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.498298 4880 generic.go:356] "Generic (PLEG): container finished" podID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerID="955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5" exitCode=0 Oct 31 16:41:59.498440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.498344 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" Oct 31 16:41:59.498440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.498374 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" event={"ID":"4ba66155-1107-4b7a-9a97-5f0ef951e1cb","Type":"ContainerDied","Data":"955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5"} Oct 31 16:41:59.498440 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.498414 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff" event={"ID":"4ba66155-1107-4b7a-9a97-5f0ef951e1cb","Type":"ContainerDied","Data":"3643468e66a0fc5c4290b738827bed52b647a753b31ef08ef322d66e21831a91"} Oct 31 16:41:59.508863 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.508846 4880 scope.go:122] "RemoveContainer" containerID="cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd" Oct 31 16:41:59.509126 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:41:59.509100 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd\": container with ID starting with cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd not found: ID does not exist" containerID="cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd" Oct 31 16:41:59.509197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.509125 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd"} err="failed to get container status \"cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd\": rpc error: code = NotFound desc = could not find container \"cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd\": container with ID starting with cb50526a13368b69b3d960a0744b5ab2bc3da1d05b915bc721f0d547de344ffd not found: ID does not exist" Oct 31 16:41:59.509197 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.509157 4880 scope.go:122] "RemoveContainer" containerID="955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5" Oct 31 16:41:59.517961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.517942 4880 scope.go:122] "RemoveContainer" containerID="955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5" Oct 31 16:41:59.518221 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:41:59.518202 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5\": container with ID starting with 955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5 not found: ID does not exist" containerID="955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5" Oct 31 16:41:59.518281 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.518227 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5"} err="failed to get container status \"955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5\": rpc error: code = NotFound desc = could not find container \"955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5\": container with ID starting with 955121ca959443ba90dcad88c2333391d783386847174a59407e8db8b9a9c8c5 not found: ID does not exist" Oct 31 16:41:59.556870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556841 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jx2x\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-kube-api-access-9jx2x\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.556870 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556867 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556878 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556888 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556897 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d7718-5378-4a24-baab-96fbaf7f8e80-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556905 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556913 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97g5\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-kube-api-access-z97g5\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556922 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556930 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d7718-5378-4a24-baab-96fbaf7f8e80-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556937 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.557041 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.556945 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba66155-1107-4b7a-9a97-5f0ef951e1cb-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:41:59.593375 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.593311 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q"] Oct 31 16:41:59.623263 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.623236 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff"] Oct 31 16:41:59.626841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.626818 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-ksjff"] Oct 31 16:41:59.685653 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.685620 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:41:59.786344 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.786305 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w"] Oct 31 16:41:59.792831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:41:59.792804 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-f43174-openshift-default-5f7c77757-bjl9w"] Oct 31 16:42:01.089825 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:01.089796 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" path="/var/lib/kubelet/pods/4ba66155-1107-4b7a-9a97-5f0ef951e1cb/volumes" Oct 31 16:42:01.090311 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:01.090296 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" path="/var/lib/kubelet/pods/f07d7718-5378-4a24-baab-96fbaf7f8e80/volumes" Oct 31 16:42:02.478960 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.478898 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" containerID="cri-o://6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66" gracePeriod=45 Oct 31 16:42:02.559300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.559267 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:02.643117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.643080 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused" Oct 31 16:42:02.735238 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.735168 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:42:02.789329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789298 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f250b3cf-92a0-421e-a50f-710da539a867-tls-certs\" (UniqueName: \"kubernetes.io/secret/f250b3cf-92a0-421e-a50f-710da539a867-tls-certs\") pod \"f250b3cf-92a0-421e-a50f-710da539a867\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " Oct 31 16:42:02.789499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789353 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f250b3cf-92a0-421e-a50f-710da539a867-kube-api-access-md2hn\" (UniqueName: \"kubernetes.io/projected/f250b3cf-92a0-421e-a50f-710da539a867-kube-api-access-md2hn\") pod \"f250b3cf-92a0-421e-a50f-710da539a867\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " Oct 31 16:42:02.789499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789433 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-tmp-dir\") pod \"f250b3cf-92a0-421e-a50f-710da539a867\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " Oct 31 16:42:02.789499 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789483 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-model-cache\") pod \"f250b3cf-92a0-421e-a50f-710da539a867\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " Oct 31 16:42:02.789678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789566 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-home\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-home\") pod \"f250b3cf-92a0-421e-a50f-710da539a867\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " Oct 31 16:42:02.789678 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789651 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-dshm\") pod \"f250b3cf-92a0-421e-a50f-710da539a867\" (UID: \"f250b3cf-92a0-421e-a50f-710da539a867\") " Oct 31 16:42:02.789792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789698 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-tmp-dir" pod "f250b3cf-92a0-421e-a50f-710da539a867" (UID: "f250b3cf-92a0-421e-a50f-710da539a867"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:02.789792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789761 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-model-cache" pod "f250b3cf-92a0-421e-a50f-710da539a867" (UID: "f250b3cf-92a0-421e-a50f-710da539a867"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:02.789873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.789794 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-home" pod "f250b3cf-92a0-421e-a50f-710da539a867" (UID: "f250b3cf-92a0-421e-a50f-710da539a867"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:02.790159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.790116 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:02.790297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.790175 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:02.790297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.790192 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:02.791658 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.791633 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f250b3cf-92a0-421e-a50f-710da539a867-tls-certs" pod "f250b3cf-92a0-421e-a50f-710da539a867" (UID: "f250b3cf-92a0-421e-a50f-710da539a867"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:02.791735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.791673 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f250b3cf-92a0-421e-a50f-710da539a867-kube-api-access-md2hn" pod "f250b3cf-92a0-421e-a50f-710da539a867" (UID: "f250b3cf-92a0-421e-a50f-710da539a867"). InnerVolumeSpecName "kube-api-access-md2hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:02.791735 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.791692 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-dshm" pod "f250b3cf-92a0-421e-a50f-710da539a867" (UID: "f250b3cf-92a0-421e-a50f-710da539a867"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:02.891805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.891774 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f250b3cf-92a0-421e-a50f-710da539a867-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:02.891805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.891801 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f250b3cf-92a0-421e-a50f-710da539a867-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:02.892054 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:02.891815 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md2hn\" (UniqueName: \"kubernetes.io/projected/f250b3cf-92a0-421e-a50f-710da539a867-kube-api-access-md2hn\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:03.527414 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.527375 4880 generic.go:356] "Generic (PLEG): container finished" podID="f250b3cf-92a0-421e-a50f-710da539a867" containerID="6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66" exitCode=0 Oct 31 16:42:03.527824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.527425 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" Oct 31 16:42:03.527824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.527463 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" event={"ID":"f250b3cf-92a0-421e-a50f-710da539a867","Type":"ContainerDied","Data":"6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66"} Oct 31 16:42:03.527824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.527508 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f" event={"ID":"f250b3cf-92a0-421e-a50f-710da539a867","Type":"ContainerDied","Data":"8d13c6e329565664e113bb83d4c65654c9aac940f4be0ec712874125dcafe560"} Oct 31 16:42:03.527824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.527527 4880 scope.go:122] "RemoveContainer" containerID="6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66" Oct 31 16:42:03.537715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.537688 4880 scope.go:122] "RemoveContainer" containerID="6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66" Oct 31 16:42:03.537990 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:03.537971 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66\": container with ID starting with 6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66 not found: ID does not exist" containerID="6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66" Oct 31 16:42:03.538048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.537999 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66"} err="failed to get container status \"6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66\": rpc error: code = NotFound desc = could not find container \"6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66\": container with ID starting with 6f9a41d00dfa84447af468b7d605eb43df8adb31969ae72b7b4f26b67841ed66 not found: ID does not exist" Oct 31 16:42:03.680547 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.680516 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f"] Oct 31 16:42:03.726529 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:03.726497 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ac1273/distinct-w0-ac1273-kserve-5b96f49fcb-vmf7f"] Oct 31 16:42:03.880579 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:03.880492 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:42:05.089673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:05.089639 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f250b3cf-92a0-421e-a50f-710da539a867" path="/var/lib/kubelet/pods/f250b3cf-92a0-421e-a50f-710da539a867/volumes" Oct 31 16:42:06.960826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:06.960791 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:07.160682 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:07.160648 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:07.468762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:07.468733 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:12.735423 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:12.735384 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4"] Oct 31 16:42:12.735958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:12.735688 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" containerID="cri-o://397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb" gracePeriod=30 Oct 31 16:42:12.761952 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:12.761875 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:14.489966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:14.489918 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:14.613673 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:14.613631 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" containerID="cri-o://1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883" gracePeriod=45 Oct 31 16:42:14.884250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:14.884225 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:42:15.026233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026196 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-model-cache\") pod \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " Oct 31 16:42:15.026432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026266 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tls-certs\") pod \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " Oct 31 16:42:15.026432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026335 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tmp-dir\") pod \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " Oct 31 16:42:15.026557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026437 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-dshm\") pod \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " Oct 31 16:42:15.026557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026497 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-model-cache" pod "40c4a6ca-af68-4383-bcb9-dd84893a0fc6" (UID: "40c4a6ca-af68-4383-bcb9-dd84893a0fc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:15.026557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026514 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-home\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-home\") pod \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " Oct 31 16:42:15.026722 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026560 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-kube-api-access-6wx5p\" (UniqueName: \"kubernetes.io/projected/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-kube-api-access-6wx5p\") pod \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\" (UID: \"40c4a6ca-af68-4383-bcb9-dd84893a0fc6\") " Oct 31 16:42:15.026835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026795 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-home" pod "40c4a6ca-af68-4383-bcb9-dd84893a0fc6" (UID: "40c4a6ca-af68-4383-bcb9-dd84893a0fc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:15.027018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.026995 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:15.027018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.027015 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:15.027263 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.027074 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tmp-dir" pod "40c4a6ca-af68-4383-bcb9-dd84893a0fc6" (UID: "40c4a6ca-af68-4383-bcb9-dd84893a0fc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:15.029040 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.028996 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-dshm" pod "40c4a6ca-af68-4383-bcb9-dd84893a0fc6" (UID: "40c4a6ca-af68-4383-bcb9-dd84893a0fc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:15.029180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.029147 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tls-certs" pod "40c4a6ca-af68-4383-bcb9-dd84893a0fc6" (UID: "40c4a6ca-af68-4383-bcb9-dd84893a0fc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:15.029413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.029384 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-kube-api-access-6wx5p" pod "40c4a6ca-af68-4383-bcb9-dd84893a0fc6" (UID: "40c4a6ca-af68-4383-bcb9-dd84893a0fc6"). InnerVolumeSpecName "kube-api-access-6wx5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:15.128650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.128625 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:15.128650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.128648 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:15.128650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.128657 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wx5p\" (UniqueName: \"kubernetes.io/projected/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-kube-api-access-6wx5p\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:15.128845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.128666 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40c4a6ca-af68-4383-bcb9-dd84893a0fc6-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:15.611481 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.611443 4880 generic.go:356] "Generic (PLEG): container finished" podID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerID="1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883" exitCode=0 Oct 31 16:42:15.611902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.611499 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" Oct 31 16:42:15.611902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.611700 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" event={"ID":"40c4a6ca-af68-4383-bcb9-dd84893a0fc6","Type":"ContainerDied","Data":"1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883"} Oct 31 16:42:15.611902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.611740 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q" event={"ID":"40c4a6ca-af68-4383-bcb9-dd84893a0fc6","Type":"ContainerDied","Data":"0c44681c632f85ba8f92463854689a8bd49b0a4273bf4875d089832fd28d216f"} Oct 31 16:42:15.611902 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.611763 4880 scope.go:122] "RemoveContainer" containerID="1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883" Oct 31 16:42:15.623313 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.623148 4880 scope.go:122] "RemoveContainer" containerID="1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883" Oct 31 16:42:15.623485 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:15.623449 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883\": container with ID starting with 1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883 not found: ID does not exist" containerID="1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883" Oct 31 16:42:15.623534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.623498 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883"} err="failed to get container status \"1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883\": rpc error: code = NotFound desc = could not find container \"1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883\": container with ID starting with 1dc414e84dd6da1442b2111a93daf22a2e11abba32778c19be932c9476600883 not found: ID does not exist" Oct 31 16:42:15.766219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.766189 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q"] Oct 31 16:42:15.815329 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:15.815295 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ef6d93/simulator-w5-ef6d93-kserve-79df45d547-smx4q"] Oct 31 16:42:17.089589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:17.089553 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" path="/var/lib/kubelet/pods/40c4a6ca-af68-4383-bcb9-dd84893a0fc6/volumes" Oct 31 16:42:17.998061 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:17.998031 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:42:18.065402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.065317 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159254 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5072770-07b2-4574-800f-b34f39fbba10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5072770-07b2-4574-800f-b34f39fbba10-istiod-ca-cert\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159313 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-certs\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159370 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-istio-token\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-istio-token\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159454 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-socket\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159492 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-data\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159519 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-credential-socket\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159565 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-kube-api-access-lwbsz\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-kube-api-access-lwbsz\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159597 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5072770-07b2-4574-800f-b34f39fbba10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5072770-07b2-4574-800f-b34f39fbba10-istio-podinfo\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159647 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-envoy\") pod \"a5072770-07b2-4574-800f-b34f39fbba10\" (UID: \"a5072770-07b2-4574-800f-b34f39fbba10\") " Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159683 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-certs" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159765 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5072770-07b2-4574-800f-b34f39fbba10-istiod-ca-cert" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:42:18.160029 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.159808 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-credential-socket" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:18.161231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.160129 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.161231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.160169 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5072770-07b2-4574-800f-b34f39fbba10-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.161231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.160184 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.161231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.160210 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-socket" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:18.161231 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.160283 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-data" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:18.162395 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.162356 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-istio-token" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:18.162660 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.162637 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-kube-api-access-lwbsz" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "kube-api-access-lwbsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:18.162960 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.162942 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-envoy" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:18.164607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.164573 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5072770-07b2-4574-800f-b34f39fbba10-istio-podinfo" pod "a5072770-07b2-4574-800f-b34f39fbba10" (UID: "a5072770-07b2-4574-800f-b34f39fbba10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:42:18.261540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.261508 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.261540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.261541 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.261540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.261554 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.261823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.261563 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwbsz\" (UniqueName: \"kubernetes.io/projected/a5072770-07b2-4574-800f-b34f39fbba10-kube-api-access-lwbsz\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.261823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.261574 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5072770-07b2-4574-800f-b34f39fbba10-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.261823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.261582 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5072770-07b2-4574-800f-b34f39fbba10-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:18.632509 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.632477 4880 generic.go:356] "Generic (PLEG): container finished" podID="a5072770-07b2-4574-800f-b34f39fbba10" containerID="397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb" exitCode=0 Oct 31 16:42:18.632692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.632521 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" Oct 31 16:42:18.632692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.632560 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" event={"ID":"a5072770-07b2-4574-800f-b34f39fbba10","Type":"ContainerDied","Data":"397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb"} Oct 31 16:42:18.632692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.632615 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4" event={"ID":"a5072770-07b2-4574-800f-b34f39fbba10","Type":"ContainerDied","Data":"d8f62f80fbbca54e1380c7176428012bda7d9ca10b1601b57ed6d8d226423f27"} Oct 31 16:42:18.632692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.632637 4880 scope.go:122] "RemoveContainer" containerID="397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb" Oct 31 16:42:18.646546 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.646525 4880 scope.go:122] "RemoveContainer" containerID="397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb" Oct 31 16:42:18.646904 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:18.646883 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb\": container with ID starting with 397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb not found: ID does not exist" containerID="397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb" Oct 31 16:42:18.646961 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.646913 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb"} err="failed to get container status \"397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb\": rpc error: code = NotFound desc = could not find container \"397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb\": container with ID starting with 397717daf083eb957842b58f0bd37ad195f83119ca6eacb32791dc986dae41eb not found: ID does not exist" Oct 31 16:42:18.785163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.785122 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4"] Oct 31 16:42:18.832432 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:18.832403 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ac1273-openshift-default-79699df87-xjtq4"] Oct 31 16:42:19.089455 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:19.089425 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5072770-07b2-4574-800f-b34f39fbba10" path="/var/lib/kubelet/pods/a5072770-07b2-4574-800f-b34f39fbba10/volumes" Oct 31 16:42:19.428048 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:19.428015 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh"] Oct 31 16:42:19.434208 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:19.434184 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td"] Oct 31 16:42:19.474076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:19.474046 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg"] Oct 31 16:42:19.603640 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:19.603600 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:23.372798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:23.372757 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:24.935161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:24.929730 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:25.108162 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:25.103636 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:25.702821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:25.702783 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:26.821216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:26.821175 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:27.022663 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:27.022630 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:28.869260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:28.869227 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:29.682936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:29.682903 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:29.829912 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:29.829876 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:31.267554 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:31.267517 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:31.428099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:31.428061 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:33.035101 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.035066 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp"] Oct 31 16:42:33.035505 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.035288 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" podUID="4a124856-37c9-4483-99c6-fa11207980cf" containerName="maas-api" containerID="cri-o://416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d" gracePeriod=30 Oct 31 16:42:33.159675 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.159629 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:33.332873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.332845 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:42:33.397619 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.397581 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls\") pod \"4a124856-37c9-4483-99c6-fa11207980cf\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " Oct 31 16:42:33.397794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.397648 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-metrics-certs\") pod \"4a124856-37c9-4483-99c6-fa11207980cf\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " Oct 31 16:42:33.397794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.397702 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a124856-37c9-4483-99c6-fa11207980cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a124856-37c9-4483-99c6-fa11207980cf-openshift-service-ca\") pod \"4a124856-37c9-4483-99c6-fa11207980cf\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " Oct 31 16:42:33.397794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.397750 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a124856-37c9-4483-99c6-fa11207980cf-kube-api-access-zc5hx\" (UniqueName: \"kubernetes.io/projected/4a124856-37c9-4483-99c6-fa11207980cf-kube-api-access-zc5hx\") pod \"4a124856-37c9-4483-99c6-fa11207980cf\" (UID: \"4a124856-37c9-4483-99c6-fa11207980cf\") " Oct 31 16:42:33.398099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.398065 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a124856-37c9-4483-99c6-fa11207980cf-openshift-service-ca" pod "4a124856-37c9-4483-99c6-fa11207980cf" (UID: "4a124856-37c9-4483-99c6-fa11207980cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:42:33.399838 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.399813 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls" pod "4a124856-37c9-4483-99c6-fa11207980cf" (UID: "4a124856-37c9-4483-99c6-fa11207980cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:33.399921 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.399837 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-metrics-certs" pod "4a124856-37c9-4483-99c6-fa11207980cf" (UID: "4a124856-37c9-4483-99c6-fa11207980cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:33.399921 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.399852 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a124856-37c9-4483-99c6-fa11207980cf-kube-api-access-zc5hx" pod "4a124856-37c9-4483-99c6-fa11207980cf" (UID: "4a124856-37c9-4483-99c6-fa11207980cf"). InnerVolumeSpecName "kube-api-access-zc5hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:33.498946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.498913 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:33.498946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.498948 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a124856-37c9-4483-99c6-fa11207980cf-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:33.499187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.498962 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a124856-37c9-4483-99c6-fa11207980cf-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:33.499187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.498977 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc5hx\" (UniqueName: \"kubernetes.io/projected/4a124856-37c9-4483-99c6-fa11207980cf-kube-api-access-zc5hx\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:33.500941 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.500919 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:42:33.508479 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.508459 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:42:33.511650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.511631 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:42:33.519498 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.519476 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:42:33.519746 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.519726 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:42:33.530412 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.530394 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:42:33.736718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.736685 4880 generic.go:356] "Generic (PLEG): container finished" podID="4a124856-37c9-4483-99c6-fa11207980cf" containerID="416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d" exitCode=0 Oct 31 16:42:33.736897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.736733 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" Oct 31 16:42:33.736897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.736771 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" event={"ID":"4a124856-37c9-4483-99c6-fa11207980cf","Type":"ContainerDied","Data":"416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d"} Oct 31 16:42:33.736897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.736852 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp" event={"ID":"4a124856-37c9-4483-99c6-fa11207980cf","Type":"ContainerDied","Data":"48598615bc5da5fff76d96f46805fb819c01450cefc4686411dfd390089bb320"} Oct 31 16:42:33.736897 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.736880 4880 scope.go:122] "RemoveContainer" containerID="416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d" Oct 31 16:42:33.747391 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.747366 4880 scope.go:122] "RemoveContainer" containerID="416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d" Oct 31 16:42:33.747686 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:33.747663 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d\": container with ID starting with 416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d not found: ID does not exist" containerID="416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d" Oct 31 16:42:33.747790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.747692 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d"} err="failed to get container status \"416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d\": rpc error: code = NotFound desc = could not find container \"416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d\": container with ID starting with 416ad2824b1ec8fa00d7ce570f8f8edb37fd1ae36167c92de1cf0ba92ead2b0d not found: ID does not exist" Oct 31 16:42:33.880008 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:33.879975 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:42:33.889428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.889383 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp"] Oct 31 16:42:33.892368 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:33.892345 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-ef6d93-6f84d56cbf-pdvgp"] Oct 31 16:42:34.194469 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.194409 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b"] Oct 31 16:42:34.194857 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.194611 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" podUID="0a301a2c-fca6-4540-8362-88bd33ff735e" containerName="maas-api" containerID="cri-o://5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4" gracePeriod=30 Oct 31 16:42:34.328317 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.328281 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:34.448826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.448710 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" containerID="cri-o://e9adf2155eab8d9e7feab4cdff98a34b3fa4a46eca3cd721b6da5916f5201109" gracePeriod=45 Oct 31 16:42:34.453448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.453412 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" containerID="cri-o://6a8a41ca613e0ad1c80d9bdcda68e21b74473cace7a4ef6bf188c1fb3fb0d5f4" gracePeriod=45 Oct 31 16:42:34.493430 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.493368 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" containerID="cri-o://b4af165fd56b353523afe4877f467109d41f3fec7e99f32fa14df453cb562fcc" gracePeriod=45 Oct 31 16:42:34.498544 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.498510 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": EOF" Oct 31 16:42:34.575823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.575769 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused" Oct 31 16:42:34.620822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.620794 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:42:34.701747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.701623 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:34.721533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.721492 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-metrics-certs\") pod \"0a301a2c-fca6-4540-8362-88bd33ff735e\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " Oct 31 16:42:34.721696 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.721569 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls\") pod \"0a301a2c-fca6-4540-8362-88bd33ff735e\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " Oct 31 16:42:34.722159 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.722107 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a301a2c-fca6-4540-8362-88bd33ff735e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a301a2c-fca6-4540-8362-88bd33ff735e-openshift-service-ca\") pod \"0a301a2c-fca6-4540-8362-88bd33ff735e\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " Oct 31 16:42:34.722286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.722234 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a301a2c-fca6-4540-8362-88bd33ff735e-kube-api-access-5w2tk\" (UniqueName: \"kubernetes.io/projected/0a301a2c-fca6-4540-8362-88bd33ff735e-kube-api-access-5w2tk\") pod \"0a301a2c-fca6-4540-8362-88bd33ff735e\" (UID: \"0a301a2c-fca6-4540-8362-88bd33ff735e\") " Oct 31 16:42:34.722787 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.722729 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a301a2c-fca6-4540-8362-88bd33ff735e-openshift-service-ca" pod "0a301a2c-fca6-4540-8362-88bd33ff735e" (UID: "0a301a2c-fca6-4540-8362-88bd33ff735e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:42:34.724625 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.724580 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-metrics-certs" pod "0a301a2c-fca6-4540-8362-88bd33ff735e" (UID: "0a301a2c-fca6-4540-8362-88bd33ff735e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:34.725822 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.725799 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls" pod "0a301a2c-fca6-4540-8362-88bd33ff735e" (UID: "0a301a2c-fca6-4540-8362-88bd33ff735e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:34.727192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.727163 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a301a2c-fca6-4540-8362-88bd33ff735e-kube-api-access-5w2tk" pod "0a301a2c-fca6-4540-8362-88bd33ff735e" (UID: "0a301a2c-fca6-4540-8362-88bd33ff735e"). InnerVolumeSpecName "kube-api-access-5w2tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:34.754834 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.754737 4880 generic.go:356] "Generic (PLEG): container finished" podID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerID="b4af165fd56b353523afe4877f467109d41f3fec7e99f32fa14df453cb562fcc" exitCode=0 Oct 31 16:42:34.754991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.754883 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" event={"ID":"f065a714-871a-44eb-be2d-1f81d40e3a2b","Type":"ContainerDied","Data":"b4af165fd56b353523afe4877f467109d41f3fec7e99f32fa14df453cb562fcc"} Oct 31 16:42:34.757130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.757082 4880 generic.go:356] "Generic (PLEG): container finished" podID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerID="e9adf2155eab8d9e7feab4cdff98a34b3fa4a46eca3cd721b6da5916f5201109" exitCode=0 Oct 31 16:42:34.757316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.757181 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" event={"ID":"7c791e49-425c-4b56-8db8-3536bafe22ac","Type":"ContainerDied","Data":"e9adf2155eab8d9e7feab4cdff98a34b3fa4a46eca3cd721b6da5916f5201109"} Oct 31 16:42:34.758753 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.758724 4880 generic.go:356] "Generic (PLEG): container finished" podID="0a301a2c-fca6-4540-8362-88bd33ff735e" containerID="5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4" exitCode=0 Oct 31 16:42:34.758867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.758780 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" Oct 31 16:42:34.758867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.758830 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" event={"ID":"0a301a2c-fca6-4540-8362-88bd33ff735e","Type":"ContainerDied","Data":"5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4"} Oct 31 16:42:34.759003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.758890 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b" event={"ID":"0a301a2c-fca6-4540-8362-88bd33ff735e","Type":"ContainerDied","Data":"89bcdca5eafcc889b029034b7e5db9d545914314fdfe6d546173b51f781ae8fa"} Oct 31 16:42:34.759003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.758926 4880 scope.go:122] "RemoveContainer" containerID="5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4" Oct 31 16:42:34.769243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.769218 4880 generic.go:356] "Generic (PLEG): container finished" podID="97fca131-9d77-45bf-ade0-43efe5862df3" containerID="6a8a41ca613e0ad1c80d9bdcda68e21b74473cace7a4ef6bf188c1fb3fb0d5f4" exitCode=0 Oct 31 16:42:34.769348 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.769279 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" event={"ID":"97fca131-9d77-45bf-ade0-43efe5862df3","Type":"ContainerDied","Data":"6a8a41ca613e0ad1c80d9bdcda68e21b74473cace7a4ef6bf188c1fb3fb0d5f4"} Oct 31 16:42:34.784561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.784520 4880 scope.go:122] "RemoveContainer" containerID="5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4" Oct 31 16:42:34.784906 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:34.784883 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4\": container with ID starting with 5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4 not found: ID does not exist" containerID="5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4" Oct 31 16:42:34.785013 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.784916 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4"} err="failed to get container status \"5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4\": rpc error: code = NotFound desc = could not find container \"5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4\": container with ID starting with 5f70c4818cdeeaafcd0eb236a81fa1f1015b9247a02bf46e78a5028fb87dcdb4 not found: ID does not exist" Oct 31 16:42:34.788377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.788360 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:42:34.813986 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.813949 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:42:34.822567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.822540 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:42:34.823013 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.822984 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-tmp-dir\") pod \"f065a714-871a-44eb-be2d-1f81d40e3a2b\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " Oct 31 16:42:34.823108 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823043 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f065a714-871a-44eb-be2d-1f81d40e3a2b-kube-api-access-hpmln\" (UniqueName: \"kubernetes.io/projected/f065a714-871a-44eb-be2d-1f81d40e3a2b-kube-api-access-hpmln\") pod \"f065a714-871a-44eb-be2d-1f81d40e3a2b\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " Oct 31 16:42:34.823194 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823179 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-model-cache\") pod \"f065a714-871a-44eb-be2d-1f81d40e3a2b\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " Oct 31 16:42:34.823266 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823228 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-tmp-dir" pod "f065a714-871a-44eb-be2d-1f81d40e3a2b" (UID: "f065a714-871a-44eb-be2d-1f81d40e3a2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.823325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823266 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-home\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-home\") pod \"f065a714-871a-44eb-be2d-1f81d40e3a2b\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " Oct 31 16:42:34.823325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823308 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f065a714-871a-44eb-be2d-1f81d40e3a2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f065a714-871a-44eb-be2d-1f81d40e3a2b-tls-certs\") pod \"f065a714-871a-44eb-be2d-1f81d40e3a2b\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " Oct 31 16:42:34.823439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823352 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-dshm\") pod \"f065a714-871a-44eb-be2d-1f81d40e3a2b\" (UID: \"f065a714-871a-44eb-be2d-1f81d40e3a2b\") " Oct 31 16:42:34.823439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823420 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-model-cache" pod "f065a714-871a-44eb-be2d-1f81d40e3a2b" (UID: "f065a714-871a-44eb-be2d-1f81d40e3a2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.823603 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823582 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-home" pod "f065a714-871a-44eb-be2d-1f81d40e3a2b" (UID: "f065a714-871a-44eb-be2d-1f81d40e3a2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.823781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823762 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.823856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823787 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.823856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823802 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a301a2c-fca6-4540-8362-88bd33ff735e-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.823856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823817 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a301a2c-fca6-4540-8362-88bd33ff735e-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.823856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823831 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w2tk\" (UniqueName: \"kubernetes.io/projected/0a301a2c-fca6-4540-8362-88bd33ff735e-kube-api-access-5w2tk\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.823856 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823845 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.824081 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.823858 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.825701 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.825669 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f065a714-871a-44eb-be2d-1f81d40e3a2b-kube-api-access-hpmln" pod "f065a714-871a-44eb-be2d-1f81d40e3a2b" (UID: "f065a714-871a-44eb-be2d-1f81d40e3a2b"). InnerVolumeSpecName "kube-api-access-hpmln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:34.826066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.826013 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-dshm" pod "f065a714-871a-44eb-be2d-1f81d40e3a2b" (UID: "f065a714-871a-44eb-be2d-1f81d40e3a2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.826324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.826305 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f065a714-871a-44eb-be2d-1f81d40e3a2b-tls-certs" pod "f065a714-871a-44eb-be2d-1f81d40e3a2b" (UID: "f065a714-871a-44eb-be2d-1f81d40e3a2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:34.911650 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.911608 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b"] Oct 31 16:42:34.915539 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.915504 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ac1273-987b8d96-zbx7b"] Oct 31 16:42:34.924428 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924400 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-model-cache\") pod \"97fca131-9d77-45bf-ade0-43efe5862df3\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " Oct 31 16:42:34.924601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924447 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-home\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-home\") pod \"7c791e49-425c-4b56-8db8-3536bafe22ac\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " Oct 31 16:42:34.924601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924474 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-dshm\") pod \"97fca131-9d77-45bf-ade0-43efe5862df3\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " Oct 31 16:42:34.924601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924491 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c791e49-425c-4b56-8db8-3536bafe22ac-kube-api-access-hvqb5\" (UniqueName: \"kubernetes.io/projected/7c791e49-425c-4b56-8db8-3536bafe22ac-kube-api-access-hvqb5\") pod \"7c791e49-425c-4b56-8db8-3536bafe22ac\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " Oct 31 16:42:34.924601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924559 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fca131-9d77-45bf-ade0-43efe5862df3-kube-api-access-txq56\" (UniqueName: \"kubernetes.io/projected/97fca131-9d77-45bf-ade0-43efe5862df3-kube-api-access-txq56\") pod \"97fca131-9d77-45bf-ade0-43efe5862df3\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " Oct 31 16:42:34.924812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924610 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-home\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-home\") pod \"97fca131-9d77-45bf-ade0-43efe5862df3\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " Oct 31 16:42:34.924812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924650 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97fca131-9d77-45bf-ade0-43efe5862df3-tls-certs\" (UniqueName: \"kubernetes.io/secret/97fca131-9d77-45bf-ade0-43efe5862df3-tls-certs\") pod \"97fca131-9d77-45bf-ade0-43efe5862df3\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " Oct 31 16:42:34.924812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924692 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-tmp-dir\") pod \"97fca131-9d77-45bf-ade0-43efe5862df3\" (UID: \"97fca131-9d77-45bf-ade0-43efe5862df3\") " Oct 31 16:42:34.924812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924725 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c791e49-425c-4b56-8db8-3536bafe22ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c791e49-425c-4b56-8db8-3536bafe22ac-tls-certs\") pod \"7c791e49-425c-4b56-8db8-3536bafe22ac\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " Oct 31 16:42:34.924812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924737 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-home" pod "7c791e49-425c-4b56-8db8-3536bafe22ac" (UID: "7c791e49-425c-4b56-8db8-3536bafe22ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.924812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924742 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-model-cache" pod "97fca131-9d77-45bf-ade0-43efe5862df3" (UID: "97fca131-9d77-45bf-ade0-43efe5862df3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.924812 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924785 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-tmp-dir\") pod \"7c791e49-425c-4b56-8db8-3536bafe22ac\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " Oct 31 16:42:34.925219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924822 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-dshm\") pod \"7c791e49-425c-4b56-8db8-3536bafe22ac\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " Oct 31 16:42:34.925219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924877 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-model-cache\") pod \"7c791e49-425c-4b56-8db8-3536bafe22ac\" (UID: \"7c791e49-425c-4b56-8db8-3536bafe22ac\") " Oct 31 16:42:34.925219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.924987 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-tmp-dir" pod "97fca131-9d77-45bf-ade0-43efe5862df3" (UID: "97fca131-9d77-45bf-ade0-43efe5862df3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.925219 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925003 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-home" pod "97fca131-9d77-45bf-ade0-43efe5862df3" (UID: "97fca131-9d77-45bf-ade0-43efe5862df3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.925438 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925280 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-tmp-dir" pod "7c791e49-425c-4b56-8db8-3536bafe22ac" (UID: "7c791e49-425c-4b56-8db8-3536bafe22ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.925517 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925493 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-model-cache" pod "7c791e49-425c-4b56-8db8-3536bafe22ac" (UID: "7c791e49-425c-4b56-8db8-3536bafe22ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.925589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925520 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f065a714-871a-44eb-be2d-1f81d40e3a2b-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.925589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925538 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.925589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925552 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.925589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925567 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.925589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925581 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpmln\" (UniqueName: \"kubernetes.io/projected/f065a714-871a-44eb-be2d-1f81d40e3a2b-kube-api-access-hpmln\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.925824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925596 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.925824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925610 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f065a714-871a-44eb-be2d-1f81d40e3a2b-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.925824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.925625 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:34.927271 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.927249 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c791e49-425c-4b56-8db8-3536bafe22ac-kube-api-access-hvqb5" pod "7c791e49-425c-4b56-8db8-3536bafe22ac" (UID: "7c791e49-425c-4b56-8db8-3536bafe22ac"). InnerVolumeSpecName "kube-api-access-hvqb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:34.927448 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.927426 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fca131-9d77-45bf-ade0-43efe5862df3-tls-certs" pod "97fca131-9d77-45bf-ade0-43efe5862df3" (UID: "97fca131-9d77-45bf-ade0-43efe5862df3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:34.927549 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.927428 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c791e49-425c-4b56-8db8-3536bafe22ac-tls-certs" pod "7c791e49-425c-4b56-8db8-3536bafe22ac" (UID: "7c791e49-425c-4b56-8db8-3536bafe22ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:34.927549 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.927523 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fca131-9d77-45bf-ade0-43efe5862df3-kube-api-access-txq56" pod "97fca131-9d77-45bf-ade0-43efe5862df3" (UID: "97fca131-9d77-45bf-ade0-43efe5862df3"). InnerVolumeSpecName "kube-api-access-txq56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:34.927800 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.927777 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-dshm" pod "7c791e49-425c-4b56-8db8-3536bafe22ac" (UID: "7c791e49-425c-4b56-8db8-3536bafe22ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.927887 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.927826 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-dshm" pod "97fca131-9d77-45bf-ade0-43efe5862df3" (UID: "97fca131-9d77-45bf-ade0-43efe5862df3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:34.982003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:34.981907 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:35.034125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.034091 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:35.034125 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.034125 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c791e49-425c-4b56-8db8-3536bafe22ac-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:35.034433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.034173 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97fca131-9d77-45bf-ade0-43efe5862df3-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:35.034433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.034190 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvqb5\" (UniqueName: \"kubernetes.io/projected/7c791e49-425c-4b56-8db8-3536bafe22ac-kube-api-access-hvqb5\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:35.034433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.034203 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txq56\" (UniqueName: \"kubernetes.io/projected/97fca131-9d77-45bf-ade0-43efe5862df3-kube-api-access-txq56\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:35.034433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.034215 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fca131-9d77-45bf-ade0-43efe5862df3-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:35.034433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.034225 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c791e49-425c-4b56-8db8-3536bafe22ac-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:35.089842 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.089811 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a301a2c-fca6-4540-8362-88bd33ff735e" path="/var/lib/kubelet/pods/0a301a2c-fca6-4540-8362-88bd33ff735e/volumes" Oct 31 16:42:35.090260 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.090247 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a124856-37c9-4483-99c6-fa11207980cf" path="/var/lib/kubelet/pods/4a124856-37c9-4483-99c6-fa11207980cf/volumes" Oct 31 16:42:35.617392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.617350 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:35.779548 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.779519 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" Oct 31 16:42:35.779836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.779522 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg" event={"ID":"f065a714-871a-44eb-be2d-1f81d40e3a2b","Type":"ContainerDied","Data":"f550e166a62f8c0c86e1500c4bbae5bb9f8ba818493fa9910f90ef364e2789a8"} Oct 31 16:42:35.779963 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.779907 4880 scope.go:122] "RemoveContainer" containerID="b4af165fd56b353523afe4877f467109d41f3fec7e99f32fa14df453cb562fcc" Oct 31 16:42:35.781875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.781720 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" event={"ID":"7c791e49-425c-4b56-8db8-3536bafe22ac","Type":"ContainerDied","Data":"53d4ecc56ff03f167c635a94b062c6d37e7783ee417c27b930eb2429d1f9f9bf"} Oct 31 16:42:35.781875 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.781802 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh" Oct 31 16:42:35.786654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.786622 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" event={"ID":"97fca131-9d77-45bf-ade0-43efe5862df3","Type":"ContainerDied","Data":"0a301013a09000bfae9c1fb2c017997c98005e3753a509d7ed94105772c6139c"} Oct 31 16:42:35.786791 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.786669 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td" Oct 31 16:42:35.794515 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.794488 4880 scope.go:122] "RemoveContainer" containerID="e9adf2155eab8d9e7feab4cdff98a34b3fa4a46eca3cd721b6da5916f5201109" Oct 31 16:42:35.805483 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.805457 4880 scope.go:122] "RemoveContainer" containerID="6a8a41ca613e0ad1c80d9bdcda68e21b74473cace7a4ef6bf188c1fb3fb0d5f4" Oct 31 16:42:35.835757 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.835711 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:35.932390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.932347 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td"] Oct 31 16:42:35.935717 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.935690 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-w6-a1685e-kserve-58c9bd959c-hd7td"] Oct 31 16:42:35.995762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:35.995722 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:36.036070 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.036032 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg"] Oct 31 16:42:36.040257 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.040229 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/simulator-w6-a1685e-kserve-8856947c5-4xrxg"] Oct 31 16:42:36.141254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.141219 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh"] Oct 31 16:42:36.143859 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.143832 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a1685e/distinct-2-w6-a1685e-kserve-6475c6f58-lxrgh"] Oct 31 16:42:36.261696 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.261606 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:36.412052 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.412018 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:36.556508 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.556422 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:36.958303 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:36.958265 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:37.090582 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:37.090550 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" path="/var/lib/kubelet/pods/7c791e49-425c-4b56-8db8-3536bafe22ac/volumes" Oct 31 16:42:37.091099 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:37.091079 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" path="/var/lib/kubelet/pods/97fca131-9d77-45bf-ade0-43efe5862df3/volumes" Oct 31 16:42:37.091557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:37.091538 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" path="/var/lib/kubelet/pods/f065a714-871a-44eb-be2d-1f81d40e3a2b/volumes" Oct 31 16:42:37.679712 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:37.679673 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:41.143325 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:41.143290 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:41.336715 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:41.336669 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:41.448533 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:41.448446 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42"] Oct 31 16:42:41.449187 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:41.449072 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" containerID="cri-o://39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765" gracePeriod=30 Oct 31 16:42:41.494452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:41.494423 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:42.710057 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:42.710021 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:42.871999 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:42.871965 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:43.072795 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:43.072705 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:43.248821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:43.248782 4880 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:42:43.248984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:43.248846 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:42:46.268144 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.268096 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:46.708555 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.708528 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:42:46.851901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.851835 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-kube-api-access-c8tj9\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-kube-api-access-c8tj9\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.851901 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.851883 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-credential-socket\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852078 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.851941 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-socket\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852132 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852084 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-envoy\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852132 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852131 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-token\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-token\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852175 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-podinfo\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852206 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-certs\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852191 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-credential-socket" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:46.852280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852240 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istiod-ca-cert\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852258 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-socket" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:46.852543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852293 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-data\") pod \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\" (UID: \"c14974db-dd36-4395-b4cc-e3ebbd7bbe16\") " Oct 31 16:42:46.852710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852653 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-certs" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:46.852710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852674 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.852710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852671 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istiod-ca-cert" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:42:46.852710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852705 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.853035 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.852912 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-data" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:46.854191 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.854162 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-kube-api-access-c8tj9" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "kube-api-access-c8tj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:46.854665 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.854632 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-token" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:46.854777 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.854744 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-envoy" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:42:46.854777 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.854758 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-podinfo" pod "c14974db-dd36-4395-b4cc-e3ebbd7bbe16" (UID: "c14974db-dd36-4395-b4cc-e3ebbd7bbe16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:42:46.859300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.859274 4880 generic.go:356] "Generic (PLEG): container finished" podID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerID="39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765" exitCode=0 Oct 31 16:42:46.859389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.859317 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" Oct 31 16:42:46.859431 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.859316 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" event={"ID":"c14974db-dd36-4395-b4cc-e3ebbd7bbe16","Type":"ContainerDied","Data":"39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765"} Oct 31 16:42:46.859470 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.859445 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42" event={"ID":"c14974db-dd36-4395-b4cc-e3ebbd7bbe16","Type":"ContainerDied","Data":"49ae6e8ed970745d2223d510bde46fe764637ef5aa610fb8832a46cfadbf14ec"} Oct 31 16:42:46.859501 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.859475 4880 scope.go:122] "RemoveContainer" containerID="39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765" Oct 31 16:42:46.877379 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.877351 4880 scope.go:122] "RemoveContainer" containerID="39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765" Oct 31 16:42:46.877701 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:46.877680 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765\": container with ID starting with 39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765 not found: ID does not exist" containerID="39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765" Oct 31 16:42:46.877790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.877709 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765"} err="failed to get container status \"39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765\": rpc error: code = NotFound desc = could not find container \"39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765\": container with ID starting with 39e6421e247c690159b0ea9e50fe411683f4c0c0b07a42a91240fd2ada44f765 not found: ID does not exist" Oct 31 16:42:46.953654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.953626 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.953654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.953651 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.953811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.953665 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.953811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.953675 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.953811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.953684 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.953811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.953693 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:46.953811 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:46.953702 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8tj9\" (UniqueName: \"kubernetes.io/projected/c14974db-dd36-4395-b4cc-e3ebbd7bbe16-kube-api-access-c8tj9\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:47.016454 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:47.016433 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42"] Oct 31 16:42:47.020120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:47.020101 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6d93-openshift-default-58df979549-d6k42"] Oct 31 16:42:47.089245 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:47.089217 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" path="/var/lib/kubelet/pods/c14974db-dd36-4395-b4cc-e3ebbd7bbe16/volumes" Oct 31 16:42:51.455651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:51.453301 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:55.043302 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.043265 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:55.235131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.235097 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:55.586711 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.586681 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5"] Oct 31 16:42:55.591160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.587444 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" podUID="c36e1157-73f5-4539-8e47-de02d4da526a" containerName="maas-api" containerID="cri-o://5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a" gracePeriod=30 Oct 31 16:42:55.714373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.714336 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:55.864291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.864264 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:42:55.952612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.952576 4880 generic.go:356] "Generic (PLEG): container finished" podID="c36e1157-73f5-4539-8e47-de02d4da526a" containerID="5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a" exitCode=0 Oct 31 16:42:55.952794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.952624 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" Oct 31 16:42:55.952794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.952657 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" event={"ID":"c36e1157-73f5-4539-8e47-de02d4da526a","Type":"ContainerDied","Data":"5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a"} Oct 31 16:42:55.952794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.952731 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5" event={"ID":"c36e1157-73f5-4539-8e47-de02d4da526a","Type":"ContainerDied","Data":"a3c563d1b432673abc1c6bddbfc8ed151b1f90e0375be07f1d500952833e94d3"} Oct 31 16:42:55.952794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.952754 4880 scope.go:122] "RemoveContainer" containerID="5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a" Oct 31 16:42:55.965096 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.964945 4880 scope.go:122] "RemoveContainer" containerID="5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a" Oct 31 16:42:55.965400 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:42:55.965383 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a\": container with ID starting with 5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a not found: ID does not exist" containerID="5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a" Oct 31 16:42:55.965467 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:55.965410 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a"} err="failed to get container status \"5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a\": rpc error: code = NotFound desc = could not find container \"5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a\": container with ID starting with 5d4cf41aa51ea9b519dc6b5d9e734e4e61df832490c53e7eb68f28f85a1c913a not found: ID does not exist" Oct 31 16:42:56.005667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.005640 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls\") pod \"c36e1157-73f5-4539-8e47-de02d4da526a\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " Oct 31 16:42:56.005843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.005684 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-metrics-certs\") pod \"c36e1157-73f5-4539-8e47-de02d4da526a\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " Oct 31 16:42:56.005916 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.005866 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36e1157-73f5-4539-8e47-de02d4da526a-kube-api-access-l4csp\" (UniqueName: \"kubernetes.io/projected/c36e1157-73f5-4539-8e47-de02d4da526a-kube-api-access-l4csp\") pod \"c36e1157-73f5-4539-8e47-de02d4da526a\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " Oct 31 16:42:56.005993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.005914 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36e1157-73f5-4539-8e47-de02d4da526a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36e1157-73f5-4539-8e47-de02d4da526a-openshift-service-ca\") pod \"c36e1157-73f5-4539-8e47-de02d4da526a\" (UID: \"c36e1157-73f5-4539-8e47-de02d4da526a\") " Oct 31 16:42:56.006626 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.006598 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36e1157-73f5-4539-8e47-de02d4da526a-openshift-service-ca" pod "c36e1157-73f5-4539-8e47-de02d4da526a" (UID: "c36e1157-73f5-4539-8e47-de02d4da526a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:42:56.009190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.009127 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-metrics-certs" pod "c36e1157-73f5-4539-8e47-de02d4da526a" (UID: "c36e1157-73f5-4539-8e47-de02d4da526a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:56.009351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.009187 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls" pod "c36e1157-73f5-4539-8e47-de02d4da526a" (UID: "c36e1157-73f5-4539-8e47-de02d4da526a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:42:56.009485 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.009460 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36e1157-73f5-4539-8e47-de02d4da526a-kube-api-access-l4csp" pod "c36e1157-73f5-4539-8e47-de02d4da526a" (UID: "c36e1157-73f5-4539-8e47-de02d4da526a"). InnerVolumeSpecName "kube-api-access-l4csp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:42:56.106938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.106863 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4csp\" (UniqueName: \"kubernetes.io/projected/c36e1157-73f5-4539-8e47-de02d4da526a-kube-api-access-l4csp\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:56.106938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.106892 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c36e1157-73f5-4539-8e47-de02d4da526a-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:56.106938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.106902 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:56.106938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.106911 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36e1157-73f5-4539-8e47-de02d4da526a-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:42:56.365667 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.365594 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5"] Oct 31 16:42:56.370202 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.370164 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a1685e-f79f75d65-92fd5"] Oct 31 16:42:56.692180 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:56.692110 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:57.089929 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:57.089854 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36e1157-73f5-4539-8e47-de02d4da526a" path="/var/lib/kubelet/pods/c36e1157-73f5-4539-8e47-de02d4da526a/volumes" Oct 31 16:42:57.333641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:57.333608 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:57.553654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:57.553617 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:42:57.695982 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:42:57.695946 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:01.444110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:01.444077 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:02.318633 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:02.318599 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:02.765826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:02.765792 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:03.883121 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:43:03.883024 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:43:07.557843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:07.557809 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:08.210781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:08.210735 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:12.352785 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:12.352755 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:13.939710 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:13.939676 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:14.162837 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:14.162802 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:14.275349 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:14.275264 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx"] Oct 31 16:43:14.275528 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:14.275505 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" containerID="cri-o://245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95" gracePeriod=30 Oct 31 16:43:14.334979 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:14.334923 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:17.171446 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:17.171415 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:19.536635 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.536611 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:43:19.643671 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643641 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-data\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.643815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643690 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-kube-api-access-pz2sk\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-kube-api-access-pz2sk\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.643815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643728 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c69a68-e687-4404-a29a-f5c441377289-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c69a68-e687-4404-a29a-f5c441377289-istiod-ca-cert\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.643815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643760 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-certs\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.643815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643801 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-envoy\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.644021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643835 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-socket\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.644021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643864 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-credential-socket\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.644021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643894 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-istio-token\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-istio-token\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.644021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643943 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-data" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:43:19.644021 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.643962 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3c69a68-e687-4404-a29a-f5c441377289-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c69a68-e687-4404-a29a-f5c441377289-istio-podinfo\") pod \"a3c69a68-e687-4404-a29a-f5c441377289\" (UID: \"a3c69a68-e687-4404-a29a-f5c441377289\") " Oct 31 16:43:19.644352 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.644332 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c69a68-e687-4404-a29a-f5c441377289-istiod-ca-cert" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:43:19.646129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.644233 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-socket" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:43:19.646129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.644509 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-credential-socket" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:43:19.646129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.644659 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-certs" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:43:19.646129 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.645971 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.646764 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.646732 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3c69a68-e687-4404-a29a-f5c441377289-istio-podinfo" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:43:19.647450 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.647413 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-kube-api-access-pz2sk" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "kube-api-access-pz2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:43:19.647552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.647461 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-istio-token" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:43:19.647552 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.647534 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-envoy" pod "a3c69a68-e687-4404-a29a-f5c441377289" (UID: "a3c69a68-e687-4404-a29a-f5c441377289"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:43:19.747228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747194 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz2sk\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-kube-api-access-pz2sk\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.747228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747224 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3c69a68-e687-4404-a29a-f5c441377289-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.747228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747234 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.747523 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747243 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.747523 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747252 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.747523 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747260 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3c69a68-e687-4404-a29a-f5c441377289-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.747523 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747267 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3c69a68-e687-4404-a29a-f5c441377289-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:19.747523 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:19.747275 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3c69a68-e687-4404-a29a-f5c441377289-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:43:20.102778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.102701 4880 generic.go:356] "Generic (PLEG): container finished" podID="a3c69a68-e687-4404-a29a-f5c441377289" containerID="245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95" exitCode=0 Oct 31 16:43:20.102778 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.102753 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" Oct 31 16:43:20.102956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.102785 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" event={"ID":"a3c69a68-e687-4404-a29a-f5c441377289","Type":"ContainerDied","Data":"245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95"} Oct 31 16:43:20.102956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.102842 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx" event={"ID":"a3c69a68-e687-4404-a29a-f5c441377289","Type":"ContainerDied","Data":"e3fcbf7dce1b6a15725d26d0e6212c3e98a266783d5ef35a18bc121262d588da"} Oct 31 16:43:20.102956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.102855 4880 scope.go:122] "RemoveContainer" containerID="245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95" Oct 31 16:43:20.113513 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.113497 4880 scope.go:122] "RemoveContainer" containerID="245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95" Oct 31 16:43:20.113774 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:43:20.113756 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95\": container with ID starting with 245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95 not found: ID does not exist" containerID="245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95" Oct 31 16:43:20.113823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.113782 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95"} err="failed to get container status \"245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95\": rpc error: code = NotFound desc = could not find container \"245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95\": container with ID starting with 245fdb61ca58b6a26718a1dd4ce025253e56e60cf26f325a83dc0a8c76e11d95 not found: ID does not exist" Oct 31 16:43:20.308111 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.308072 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx"] Oct 31 16:43:20.349912 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:20.349880 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1685e-openshift-default-684b58587c-8ncqx"] Oct 31 16:43:21.089909 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:21.089874 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c69a68-e687-4404-a29a-f5c441377289" path="/var/lib/kubelet/pods/a3c69a68-e687-4404-a29a-f5c441377289/volumes" Oct 31 16:43:22.162604 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:22.162544 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:24.988831 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:24.988788 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:25.206567 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:25.206518 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:26.873783 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:26.873744 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:31.670323 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:31.670280 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:33.880231 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:43:33.880190 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:43:36.458577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:36.458539 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:36.852518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:36.852432 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:41.354503 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:41.354461 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:46.164568 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:46.164486 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:50.967585 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:50.967546 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:43:55.756540 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:43:55.756500 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:00.591034 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:00.590996 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:03.879969 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:44:03.879937 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:44:05.440214 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:05.440166 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:10.256886 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:10.256852 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:15.049392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:15.049352 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:19.893885 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:19.893846 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:24.681424 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:24.681390 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:27.451478 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:27.451444 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:29.474557 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:29.474522 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:31.718772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:31.718730 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:33.880608 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:44:33.880566 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:44:34.253750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:34.253717 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:37.174116 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:37.174077 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:37.374435 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:37.374400 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:39.169956 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:39.169916 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:44.083589 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:44.083551 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:48.979824 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:48.979782 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:53.981120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:53.981081 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:44:58.982030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:44:58.981991 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:00.162543 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.162506 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs"] Oct 31 16:45:00.163109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163089 4880 cpu_manager.go:414] "containerMap: removing container" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163110 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163126 4880 cpu_manager.go:414] "containerMap: removing container" podUID="0a301a2c-fca6-4540-8362-88bd33ff735e" containerName="maas-api" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163160 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a301a2c-fca6-4540-8362-88bd33ff735e" containerName="maas-api" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163182 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163191 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163211 4880 cpu_manager.go:414] "containerMap: removing container" podUID="c36e1157-73f5-4539-8e47-de02d4da526a" containerName="maas-api" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163217 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36e1157-73f5-4539-8e47-de02d4da526a" containerName="maas-api" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163227 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163233 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" Oct 31 16:45:00.163246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163246 4880 cpu_manager.go:414] "containerMap: removing container" podUID="4a124856-37c9-4483-99c6-fa11207980cf" containerName="maas-api" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163253 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a124856-37c9-4483-99c6-fa11207980cf" containerName="maas-api" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163268 4880 cpu_manager.go:414] "containerMap: removing container" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163275 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163285 4880 cpu_manager.go:414] "containerMap: removing container" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163293 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163302 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163309 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163322 4880 cpu_manager.go:414] "containerMap: removing container" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163329 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163349 4880 cpu_manager.go:414] "containerMap: removing container" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163356 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163370 4880 cpu_manager.go:414] "containerMap: removing container" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163377 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163388 4880 cpu_manager.go:414] "containerMap: removing container" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163395 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163408 4880 cpu_manager.go:414] "containerMap: removing container" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163415 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163553 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5072770-07b2-4574-800f-b34f39fbba10" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163567 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c791e49-425c-4b56-8db8-3536bafe22ac" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163577 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ca0f4d-299b-4495-a6aa-73a6e3aedae3" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163591 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fca131-9d77-45bf-ade0-43efe5862df3" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163601 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a301a2c-fca6-4540-8362-88bd33ff735e" containerName="maas-api" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163615 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f065a714-871a-44eb-be2d-1f81d40e3a2b" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163625 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36e1157-73f5-4539-8e47-de02d4da526a" containerName="maas-api" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163633 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07d7718-5378-4a24-baab-96fbaf7f8e80" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163643 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a124856-37c9-4483-99c6-fa11207980cf" containerName="maas-api" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163652 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c4a6ca-af68-4383-bcb9-dd84893a0fc6" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163662 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c69a68-e687-4404-a29a-f5c441377289" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163671 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f250b3cf-92a0-421e-a50f-710da539a867" containerName="main" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163680 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14974db-dd36-4395-b4cc-e3ebbd7bbe16" containerName="istio-proxy" Oct 31 16:45:00.163749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.163690 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba66155-1107-4b7a-9a97-5f0ef951e1cb" containerName="istio-proxy" Oct 31 16:45:00.167131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.167112 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm"] Oct 31 16:45:00.167243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.167226 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.171324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.171304 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.215343 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.215324 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6c28n\"" Oct 31 16:45:00.236074 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.236050 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpq97\" (UniqueName: \"kubernetes.io/projected/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-kube-api-access-mpq97\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.236236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.236091 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede67c-6fd3-4635-8612-8e98b18a89a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.236236 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.236121 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznmm\" (UniqueName: \"kubernetes.io/projected/62ede67c-6fd3-4635-8612-8e98b18a89a0-kube-api-access-vznmm\") pod \"maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.236330 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.236265 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.266017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.265981 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs"] Oct 31 16:45:00.270174 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.270127 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm"] Oct 31 16:45:00.337251 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.337220 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede67c-6fd3-4635-8612-8e98b18a89a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.337444 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.337276 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznmm\" (UniqueName: \"kubernetes.io/projected/62ede67c-6fd3-4635-8612-8e98b18a89a0-kube-api-access-vznmm\") pod \"maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.337444 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.337305 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.337444 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.337407 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpq97\" (UniqueName: \"kubernetes.io/projected/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-kube-api-access-mpq97\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.337945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.337925 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede67c-6fd3-4635-8612-8e98b18a89a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.338120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.338097 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.390790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.390761 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznmm\" (UniqueName: \"kubernetes.io/projected/62ede67c-6fd3-4635-8612-8e98b18a89a0-kube-api-access-vznmm\") pod \"maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.391045 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.391024 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpq97\" (UniqueName: \"kubernetes.io/projected/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-kube-api-access-mpq97\") pod \"maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.481864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.481779 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:00.488061 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.488036 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:00.862570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.862486 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs"] Oct 31 16:45:00.869038 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:45:00.869007 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ede67c_6fd3_4635_8612_8e98b18a89a0.slice/crio-ea7f900bdf12767b02253205cd9e3025c625bb2d09299e34b8eac5ed0653e9fb WatchSource:0}: Error finding container ea7f900bdf12767b02253205cd9e3025c625bb2d09299e34b8eac5ed0653e9fb: Status 404 returned error can't find the container with id ea7f900bdf12767b02253205cd9e3025c625bb2d09299e34b8eac5ed0653e9fb Oct 31 16:45:00.870958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.870940 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:45:00.913692 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:00.913660 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm"] Oct 31 16:45:00.919548 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:45:00.919518 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2e60ba_2d45_4fa0_ae0f_1ab5ab719d34.slice/crio-714b69d3683ae23260f5356ab542b21d637e4aba4adfe338fa5ac4d5a9a24bba WatchSource:0}: Error finding container 714b69d3683ae23260f5356ab542b21d637e4aba4adfe338fa5ac4d5a9a24bba: Status 404 returned error can't find the container with id 714b69d3683ae23260f5356ab542b21d637e4aba4adfe338fa5ac4d5a9a24bba Oct 31 16:45:01.715932 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:01.715894 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" event={"ID":"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34","Type":"ContainerStarted","Data":"714b69d3683ae23260f5356ab542b21d637e4aba4adfe338fa5ac4d5a9a24bba"} Oct 31 16:45:01.717234 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:01.717123 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" event={"ID":"62ede67c-6fd3-4635-8612-8e98b18a89a0","Type":"ContainerStarted","Data":"ea7f900bdf12767b02253205cd9e3025c625bb2d09299e34b8eac5ed0653e9fb"} Oct 31 16:45:03.864791 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:03.864750 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:03.896803 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:03.895951 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:45:05.746381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:05.746348 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" event={"ID":"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34","Type":"ContainerStarted","Data":"375e2acac0c582d3f26ac2e95dbacf809cf02838eedd4c693c4e1ec714fd41a2"} Oct 31 16:45:05.747923 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:05.747897 4880 generic.go:356] "Generic (PLEG): container finished" podID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerID="188e14bfe6839692132d4f1ca42cdae721ed278b22fb647b68907596154fefff" exitCode=60 Oct 31 16:45:05.748037 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:05.747973 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" event={"ID":"62ede67c-6fd3-4635-8612-8e98b18a89a0","Type":"ContainerDied","Data":"188e14bfe6839692132d4f1ca42cdae721ed278b22fb647b68907596154fefff"} Oct 31 16:45:05.748213 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:05.748199 4880 scope.go:122] "RemoveContainer" containerID="188e14bfe6839692132d4f1ca42cdae721ed278b22fb647b68907596154fefff" Oct 31 16:45:05.851177 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:05.851098 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" podStartSLOduration=1.831164936 podStartE2EDuration="5.85108465s" totalImagesPullingTime="4.019919714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:45:05.850169466 +0000 UTC m=+2552.885464265" watchObservedRunningTime="2026-10-31 16:45:05.85108465 +0000 UTC m=+2552.886379454" Oct 31 16:45:06.754750 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:06.754709 4880 generic.go:356] "Generic (PLEG): container finished" podID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerID="375e2acac0c582d3f26ac2e95dbacf809cf02838eedd4c693c4e1ec714fd41a2" exitCode=60 Oct 31 16:45:06.755254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:06.754817 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" event={"ID":"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34","Type":"ContainerDied","Data":"375e2acac0c582d3f26ac2e95dbacf809cf02838eedd4c693c4e1ec714fd41a2"} Oct 31 16:45:06.755254 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:06.755101 4880 scope.go:122] "RemoveContainer" containerID="375e2acac0c582d3f26ac2e95dbacf809cf02838eedd4c693c4e1ec714fd41a2" Oct 31 16:45:06.756946 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:06.756922 4880 generic.go:356] "Generic (PLEG): container finished" podID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerID="a5baae72d348279cb3c651b50996c56d167c6cc052d89adf89fb6ffb4fe77dc4" exitCode=60 Oct 31 16:45:06.757072 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:06.757003 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" event={"ID":"62ede67c-6fd3-4635-8612-8e98b18a89a0","Type":"ContainerDied","Data":"a5baae72d348279cb3c651b50996c56d167c6cc052d89adf89fb6ffb4fe77dc4"} Oct 31 16:45:06.757072 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:06.757062 4880 scope.go:122] "RemoveContainer" containerID="188e14bfe6839692132d4f1ca42cdae721ed278b22fb647b68907596154fefff" Oct 31 16:45:06.757381 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:06.757316 4880 scope.go:122] "RemoveContainer" containerID="a5baae72d348279cb3c651b50996c56d167c6cc052d89adf89fb6ffb4fe77dc4" Oct 31 16:45:06.757577 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:06.757557 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs_odh-ai-gateway-infra(62ede67c-6fd3-4635-8612-8e98b18a89a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" Oct 31 16:45:07.765030 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:07.764989 4880 generic.go:356] "Generic (PLEG): container finished" podID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerID="ad2ec225e2dad7bf9afc12e60f8f4e7d948bda1fd449e6226cd3338ee74bacb0" exitCode=60 Oct 31 16:45:07.765532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:07.765086 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" event={"ID":"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34","Type":"ContainerDied","Data":"ad2ec225e2dad7bf9afc12e60f8f4e7d948bda1fd449e6226cd3338ee74bacb0"} Oct 31 16:45:07.765532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:07.765162 4880 scope.go:122] "RemoveContainer" containerID="375e2acac0c582d3f26ac2e95dbacf809cf02838eedd4c693c4e1ec714fd41a2" Oct 31 16:45:07.765532 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:07.765425 4880 scope.go:122] "RemoveContainer" containerID="ad2ec225e2dad7bf9afc12e60f8f4e7d948bda1fd449e6226cd3338ee74bacb0" Oct 31 16:45:07.765740 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:07.765716 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm_odh-ai-gateway-infra(bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" Oct 31 16:45:07.767110 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:07.767093 4880 scope.go:122] "RemoveContainer" containerID="a5baae72d348279cb3c651b50996c56d167c6cc052d89adf89fb6ffb4fe77dc4" Oct 31 16:45:07.767337 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:07.767319 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs_odh-ai-gateway-infra(62ede67c-6fd3-4635-8612-8e98b18a89a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" Oct 31 16:45:08.758086 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:08.758048 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:08.774121 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:08.774091 4880 scope.go:122] "RemoveContainer" containerID="ad2ec225e2dad7bf9afc12e60f8f4e7d948bda1fd449e6226cd3338ee74bacb0" Oct 31 16:45:08.774490 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:08.774375 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm_odh-ai-gateway-infra(bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" Oct 31 16:45:13.649749 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:13.649665 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:18.083177 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:18.083119 4880 scope.go:122] "RemoveContainer" containerID="ad2ec225e2dad7bf9afc12e60f8f4e7d948bda1fd449e6226cd3338ee74bacb0" Oct 31 16:45:18.083654 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:18.083240 4880 scope.go:122] "RemoveContainer" containerID="a5baae72d348279cb3c651b50996c56d167c6cc052d89adf89fb6ffb4fe77dc4" Oct 31 16:45:18.602358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:18.602259 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:18.834118 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:18.834081 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" event={"ID":"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34","Type":"ContainerStarted","Data":"becdecdaa502b251939327d3ec841982756be5cf7137f22014f50f3c85082ec0"} Oct 31 16:45:18.836024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:18.835999 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" event={"ID":"62ede67c-6fd3-4635-8612-8e98b18a89a0","Type":"ContainerStarted","Data":"6bba21941ef8393cd06bcc27f64860b1763a3ff05cc2341dc21f1ab4b9406f05"} Oct 31 16:45:19.045335 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.045284 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" podStartSLOduration=14.982945054 podStartE2EDuration="19.045270312s" totalImagesPullingTime="4.062325258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:45:19.043291642 +0000 UTC m=+2566.078586445" watchObservedRunningTime="2026-10-31 16:45:19.045270312 +0000 UTC m=+2566.080565128" Oct 31 16:45:19.843203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.843166 4880 generic.go:356] "Generic (PLEG): container finished" podID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerID="becdecdaa502b251939327d3ec841982756be5cf7137f22014f50f3c85082ec0" exitCode=60 Oct 31 16:45:19.843728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.843222 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" event={"ID":"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34","Type":"ContainerDied","Data":"becdecdaa502b251939327d3ec841982756be5cf7137f22014f50f3c85082ec0"} Oct 31 16:45:19.843728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.843281 4880 scope.go:122] "RemoveContainer" containerID="ad2ec225e2dad7bf9afc12e60f8f4e7d948bda1fd449e6226cd3338ee74bacb0" Oct 31 16:45:19.843728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.843575 4880 scope.go:122] "RemoveContainer" containerID="becdecdaa502b251939327d3ec841982756be5cf7137f22014f50f3c85082ec0" Oct 31 16:45:19.843863 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:19.843822 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm_odh-ai-gateway-infra(bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" Oct 31 16:45:19.845308 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.845288 4880 generic.go:356] "Generic (PLEG): container finished" podID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerID="6bba21941ef8393cd06bcc27f64860b1763a3ff05cc2341dc21f1ab4b9406f05" exitCode=60 Oct 31 16:45:19.845401 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.845316 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" event={"ID":"62ede67c-6fd3-4635-8612-8e98b18a89a0","Type":"ContainerDied","Data":"6bba21941ef8393cd06bcc27f64860b1763a3ff05cc2341dc21f1ab4b9406f05"} Oct 31 16:45:19.845558 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.845546 4880 scope.go:122] "RemoveContainer" containerID="6bba21941ef8393cd06bcc27f64860b1763a3ff05cc2341dc21f1ab4b9406f05" Oct 31 16:45:19.845740 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:19.845725 4880 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs_odh-ai-gateway-infra(62ede67c-6fd3-4635-8612-8e98b18a89a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" Oct 31 16:45:19.854991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.854975 4880 scope.go:122] "RemoveContainer" containerID="a5baae72d348279cb3c651b50996c56d167c6cc052d89adf89fb6ffb4fe77dc4" Oct 31 16:45:19.996674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:19.996645 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm"] Oct 31 16:45:20.053358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:20.053328 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs"] Oct 31 16:45:20.994066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:20.994039 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:21.029984 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.029962 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:21.047903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.047878 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-kube-api-access-mpq97\" (UniqueName: \"kubernetes.io/projected/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-kube-api-access-mpq97\") pod \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " Oct 31 16:45:21.048003 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.047980 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-maas-api-service-ca\") pod \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\" (UID: \"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34\") " Oct 31 16:45:21.048318 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.048296 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-maas-api-service-ca" pod "bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" (UID: "bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:45:21.049862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.049843 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-kube-api-access-mpq97" pod "bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" (UID: "bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34"). InnerVolumeSpecName "kube-api-access-mpq97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:45:21.148444 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.148416 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ede67c-6fd3-4635-8612-8e98b18a89a0-kube-api-access-vznmm\" (UniqueName: \"kubernetes.io/projected/62ede67c-6fd3-4635-8612-8e98b18a89a0-kube-api-access-vznmm\") pod \"62ede67c-6fd3-4635-8612-8e98b18a89a0\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " Oct 31 16:45:21.148597 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.148586 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ede67c-6fd3-4635-8612-8e98b18a89a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede67c-6fd3-4635-8612-8e98b18a89a0-maas-api-service-ca\") pod \"62ede67c-6fd3-4635-8612-8e98b18a89a0\" (UID: \"62ede67c-6fd3-4635-8612-8e98b18a89a0\") " Oct 31 16:45:21.148794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.148773 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-maas-api-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:45:21.148794 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.148790 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpq97\" (UniqueName: \"kubernetes.io/projected/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34-kube-api-access-mpq97\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:45:21.148966 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.148939 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ede67c-6fd3-4635-8612-8e98b18a89a0-maas-api-service-ca" pod "62ede67c-6fd3-4635-8612-8e98b18a89a0" (UID: "62ede67c-6fd3-4635-8612-8e98b18a89a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:45:21.150573 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.150553 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ede67c-6fd3-4635-8612-8e98b18a89a0-kube-api-access-vznmm" pod "62ede67c-6fd3-4635-8612-8e98b18a89a0" (UID: "62ede67c-6fd3-4635-8612-8e98b18a89a0"). InnerVolumeSpecName "kube-api-access-vznmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:45:21.249836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.249804 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ede67c-6fd3-4635-8612-8e98b18a89a0-maas-api-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:45:21.249836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.249831 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznmm\" (UniqueName: \"kubernetes.io/projected/62ede67c-6fd3-4635-8612-8e98b18a89a0-kube-api-access-vznmm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:45:21.860983 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.860952 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" Oct 31 16:45:21.861229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.860955 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm" event={"ID":"bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34","Type":"ContainerDied","Data":"714b69d3683ae23260f5356ab542b21d637e4aba4adfe338fa5ac4d5a9a24bba"} Oct 31 16:45:21.861229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.861112 4880 scope.go:122] "RemoveContainer" containerID="becdecdaa502b251939327d3ec841982756be5cf7137f22014f50f3c85082ec0" Oct 31 16:45:21.862389 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.862370 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" Oct 31 16:45:21.862452 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.862387 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs" event={"ID":"62ede67c-6fd3-4635-8612-8e98b18a89a0","Type":"ContainerDied","Data":"ea7f900bdf12767b02253205cd9e3025c625bb2d09299e34b8eac5ed0653e9fb"} Oct 31 16:45:21.870387 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:21.870370 4880 scope.go:122] "RemoveContainer" containerID="6bba21941ef8393cd06bcc27f64860b1763a3ff05cc2341dc21f1ab4b9406f05" Oct 31 16:45:22.012026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:22.011999 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm"] Oct 31 16:45:22.057748 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:22.057721 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f7821a-29891085-59qtm"] Oct 31 16:45:22.158117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:22.158088 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs"] Oct 31 16:45:22.163259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:22.163235 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-66db1c-29891085-62jzs"] Oct 31 16:45:23.089534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:23.089503 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" path="/var/lib/kubelet/pods/62ede67c-6fd3-4635-8612-8e98b18a89a0/volumes" Oct 31 16:45:23.089911 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:23.089876 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" path="/var/lib/kubelet/pods/bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34/volumes" Oct 31 16:45:23.635730 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:23.635696 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:28.459364 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:28.459325 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:33.360516 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:33.360471 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:33.881015 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:45:33.880974 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:45:37.519189 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:37.519129 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:38.356031 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:38.355994 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:43.259954 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:43.259913 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:48.153027 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:48.152982 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:53.062336 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:53.062299 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:45:58.063128 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:45:58.063095 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:02.966200 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:02.966165 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:03.880266 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:46:03.880230 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:46:07.859565 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:07.859532 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:08.075855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:08.075813 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:12.772740 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:12.772699 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:17.772361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:17.772317 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:22.669399 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:22.669365 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:24.505823 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:24.505784 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:27.570683 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:27.570645 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:31.627324 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:31.627290 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:32.597516 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:32.597476 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:33.880372 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:46:33.880333 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:46:37.868307 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:37.868260 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:38.062333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:38.062298 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:42.958373 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:42.958271 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:43.884649 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:43.884616 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:44.022928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:44.022901 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:47.857300 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:47.857261 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:52.778588 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:52.778544 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:57.091570 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:57.091530 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:58.730464 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:58.730428 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:58.962338 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:58.962303 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:59.102058 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:59.101972 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:46:59.755369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:46:59.755333 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:03.880909 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:47:03.880874 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:47:04.175455 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:04.175421 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:05.639377 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:05.639337 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:05.786314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:05.786280 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:05.961094 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:05.960976 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:09.348907 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:09.348876 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:14.575607 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:14.575577 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:18.531534 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:18.531495 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:18.710402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:18.710367 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:19.476922 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:19.476886 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:24.364404 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:24.364370 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:28.811775 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:28.811733 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:30.291801 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:30.291760 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:30.566840 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:30.566757 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:30.712333 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:30.712262 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:31.213950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:31.213908 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:33.544600 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:33.544571 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:47:33.553246 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:33.553211 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:47:33.559278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:33.559252 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:47:33.564939 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:33.564916 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:47:33.566912 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:33.566891 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:47:33.578566 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:33.578543 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:47:33.880916 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:47:33.880838 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:47:35.875082 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:35.875043 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:36.947286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:36.947242 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:37.137948 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:37.137911 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:37.286985 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:37.286888 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:37.725015 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:37.724979 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:38.203131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:38.203075 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:38.399272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:38.399233 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:41.223733 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:41.223692 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:42.676506 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:42.676438 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg"] Oct 31 16:47:42.721043 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:42.721010 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp"] Oct 31 16:47:42.777342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:42.777303 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:46.466488 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:46.466450 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:51.510158 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:51.510085 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:56.362873 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:56.362837 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:47:57.697598 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:57.697560 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" containerID="cri-o://cdf58fc26c6e63791932aff77282162091391429802ce6611e8825f06b4b8e3a" gracePeriod=45 Oct 31 16:47:57.741026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:57.740982 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" containerID="cri-o://f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac" gracePeriod=45 Oct 31 16:47:57.819456 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:57.819427 4880 generic.go:356] "Generic (PLEG): container finished" podID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerID="cdf58fc26c6e63791932aff77282162091391429802ce6611e8825f06b4b8e3a" exitCode=0 Oct 31 16:47:57.819604 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:57.819497 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" event={"ID":"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c","Type":"ContainerDied","Data":"cdf58fc26c6e63791932aff77282162091391429802ce6611e8825f06b4b8e3a"} Oct 31 16:47:57.994803 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:57.994781 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:47:58.017846 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.017824 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:47:58.046808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.046784 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tmp-dir\") pod \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " Oct 31 16:47:58.046936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.046833 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-model-cache\") pod \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " Oct 31 16:47:58.046936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.046875 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-home\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-home\") pod \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " Oct 31 16:47:58.046936 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.046912 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-dshm\") pod \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " Oct 31 16:47:58.047109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.046945 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tls-certs\") pod \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " Oct 31 16:47:58.047109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.046973 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tls-certs\") pod \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " Oct 31 16:47:58.047109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047046 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-kube-api-access-plpqr\" (UniqueName: \"kubernetes.io/projected/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-kube-api-access-plpqr\") pod \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " Oct 31 16:47:58.047109 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047057 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tmp-dir" pod "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" (UID: "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.047345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047159 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-model-cache" pod "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" (UID: "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.047345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047164 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-home" pod "cd4f876b-52f7-41af-bd1f-63ff7ccc42be" (UID: "cd4f876b-52f7-41af-bd1f-63ff7ccc42be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.047345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047217 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tmp-dir\") pod \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " Oct 31 16:47:58.047345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047285 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-model-cache\") pod \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\" (UID: \"cd4f876b-52f7-41af-bd1f-63ff7ccc42be\") " Oct 31 16:47:58.047345 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047338 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-kube-api-access-5scjb\" (UniqueName: \"kubernetes.io/projected/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-kube-api-access-5scjb\") pod \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " Oct 31 16:47:58.047614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047414 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-home\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-home\") pod \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " Oct 31 16:47:58.047614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047465 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tmp-dir" pod "cd4f876b-52f7-41af-bd1f-63ff7ccc42be" (UID: "cd4f876b-52f7-41af-bd1f-63ff7ccc42be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.047614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047516 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-dshm\") pod \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\" (UID: \"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c\") " Oct 31 16:47:58.047614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047585 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-model-cache" pod "cd4f876b-52f7-41af-bd1f-63ff7ccc42be" (UID: "cd4f876b-52f7-41af-bd1f-63ff7ccc42be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.048370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047916 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.048370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047942 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.048370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047956 4880 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tmp-dir\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.048370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047970 4880 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-model-cache\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.048370 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.047983 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.049398 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.049357 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-home" pod "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" (UID: "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.049550 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.049515 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-dshm" pod "cd4f876b-52f7-41af-bd1f-63ff7ccc42be" (UID: "cd4f876b-52f7-41af-bd1f-63ff7ccc42be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.049662 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.049628 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-kube-api-access-plpqr" pod "cd4f876b-52f7-41af-bd1f-63ff7ccc42be" (UID: "cd4f876b-52f7-41af-bd1f-63ff7ccc42be"). InnerVolumeSpecName "kube-api-access-plpqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:47:58.049737 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.049672 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tls-certs" pod "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" (UID: "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:47:58.050763 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.050738 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tls-certs" pod "cd4f876b-52f7-41af-bd1f-63ff7ccc42be" (UID: "cd4f876b-52f7-41af-bd1f-63ff7ccc42be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:47:58.050993 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.050974 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-kube-api-access-5scjb" pod "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" (UID: "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c"). InnerVolumeSpecName "kube-api-access-5scjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:47:58.051298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.051275 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-dshm" pod "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" (UID: "d9c36c9a-01b5-42eb-9ae9-23717f85bd8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:47:58.148792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.148762 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.148792 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.148792 4880 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-dshm\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.149039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.148804 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.149039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.148816 4880 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-tls-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.149039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.148832 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plpqr\" (UniqueName: \"kubernetes.io/projected/cd4f876b-52f7-41af-bd1f-63ff7ccc42be-kube-api-access-plpqr\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.149039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.148846 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scjb\" (UniqueName: \"kubernetes.io/projected/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-kube-api-access-5scjb\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.149039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.148858 4880 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c-home\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:47:58.827316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.827275 4880 generic.go:356] "Generic (PLEG): container finished" podID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerID="f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac" exitCode=0 Oct 31 16:47:58.827781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.827333 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" Oct 31 16:47:58.827781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.827355 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" event={"ID":"cd4f876b-52f7-41af-bd1f-63ff7ccc42be","Type":"ContainerDied","Data":"f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac"} Oct 31 16:47:58.827781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.827402 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp" event={"ID":"cd4f876b-52f7-41af-bd1f-63ff7ccc42be","Type":"ContainerDied","Data":"5dc634283eee97483f5eb1adbed074673a85ff7f684e4285c776342e2a134675"} Oct 31 16:47:58.827781 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.827426 4880 scope.go:122] "RemoveContainer" containerID="f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac" Oct 31 16:47:58.828968 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.828937 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" event={"ID":"d9c36c9a-01b5-42eb-9ae9-23717f85bd8c","Type":"ContainerDied","Data":"329a0411bcf49d82a3849b7366c174390873a416a1364e28362a93f26351d058"} Oct 31 16:47:58.829068 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.829038 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg" Oct 31 16:47:58.839518 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.839496 4880 scope.go:122] "RemoveContainer" containerID="f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac" Oct 31 16:47:58.839823 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:47:58.839800 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac\": container with ID starting with f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac not found: ID does not exist" containerID="f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac" Oct 31 16:47:58.839910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.839832 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac"} err="failed to get container status \"f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac\": rpc error: code = NotFound desc = could not find container \"f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac\": container with ID starting with f7de4043a5ce784c19eda8ec66bea054a9be89048959032604ab18fb961673ac not found: ID does not exist" Oct 31 16:47:58.839910 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.839849 4880 scope.go:122] "RemoveContainer" containerID="cdf58fc26c6e63791932aff77282162091391429802ce6611e8825f06b4b8e3a" Oct 31 16:47:58.871009 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:47:58.870978 4880 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c36c9a_01b5_42eb_9ae9_23717f85bd8c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd4f876b_52f7_41af_bd1f_63ff7ccc42be.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c36c9a_01b5_42eb_9ae9_23717f85bd8c.slice/crio-329a0411bcf49d82a3849b7366c174390873a416a1364e28362a93f26351d058\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd4f876b_52f7_41af_bd1f_63ff7ccc42be.slice/crio-5dc634283eee97483f5eb1adbed074673a85ff7f684e4285c776342e2a134675\": RecentStats: unable to find data in memory cache]" Oct 31 16:47:58.979790 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:58.979707 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp"] Oct 31 16:47:59.025209 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:59.025179 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/simulator-w3-f7821a-kserve-779686fb7-pfqwp"] Oct 31 16:47:59.091169 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:59.091121 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" path="/var/lib/kubelet/pods/cd4f876b-52f7-41af-bd1f-63ff7ccc42be/volumes" Oct 31 16:47:59.091433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:59.091420 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg"] Oct 31 16:47:59.091476 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:47:59.091447 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f7821a/premium-w3-f7821a-kserve-5844fd97bb-jk5lg"] Oct 31 16:48:01.089184 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:01.089156 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" path="/var/lib/kubelet/pods/d9c36c9a-01b5-42eb-9ae9-23717f85bd8c/volumes" Oct 31 16:48:01.352026 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:01.351944 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:03.882219 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:48:03.882177 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:48:06.459990 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:06.459959 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:11.456950 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:11.456915 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:16.368864 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:16.368788 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:17.683243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:17.683202 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq"] Oct 31 16:48:17.683669 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:17.683397 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" containerID="cri-o://8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5" gracePeriod=30 Oct 31 16:48:20.134060 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:20.134026 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:20.270538 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:20.270500 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:22.934761 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.934733 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:48:22.974120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.974027 4880 generic.go:356] "Generic (PLEG): container finished" podID="f7b228db-d145-47dd-8609-76d77564749e" containerID="8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5" exitCode=0 Oct 31 16:48:22.974120 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.974076 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" Oct 31 16:48:22.974390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.974121 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" event={"ID":"f7b228db-d145-47dd-8609-76d77564749e","Type":"ContainerDied","Data":"8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5"} Oct 31 16:48:22.974390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.974195 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq" event={"ID":"f7b228db-d145-47dd-8609-76d77564749e","Type":"ContainerDied","Data":"d00b7a6a4abfa8dbd7bc8f3f3a6c1fdfaa99d90610550cd009a6e06a2536db90"} Oct 31 16:48:22.974390 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.974227 4880 scope.go:122] "RemoveContainer" containerID="8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5" Oct 31 16:48:22.984903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.984881 4880 scope.go:122] "RemoveContainer" containerID="8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5" Oct 31 16:48:22.985230 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:48:22.985207 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5\": container with ID starting with 8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5 not found: ID does not exist" containerID="8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5" Oct 31 16:48:22.985316 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:22.985257 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5"} err="failed to get container status \"8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5\": rpc error: code = NotFound desc = could not find container \"8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5\": container with ID starting with 8df9ae4e6ef913150ee9d591933ce2f062c3de449e60a271e45724b7336e2ac5 not found: ID does not exist" Oct 31 16:48:23.090213 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090174 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7b228db-d145-47dd-8609-76d77564749e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b228db-d145-47dd-8609-76d77564749e-istio-podinfo\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090244 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b228db-d145-47dd-8609-76d77564749e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b228db-d145-47dd-8609-76d77564749e-istiod-ca-cert\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090311 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-kube-api-access-dfwng\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-kube-api-access-dfwng\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090377 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-data\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090467 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-certs\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090492 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-istio-token\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-istio-token\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090523 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-credential-socket\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090563 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-envoy\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.090674 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090591 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-socket\") pod \"f7b228db-d145-47dd-8609-76d77564749e\" (UID: \"f7b228db-d145-47dd-8609-76d77564749e\") " Oct 31 16:48:23.091862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090727 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b228db-d145-47dd-8609-76d77564749e-istiod-ca-cert" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:48:23.091862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.090915 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7b228db-d145-47dd-8609-76d77564749e-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.091862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.091010 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-socket" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:23.093097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.092630 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-certs" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:23.093097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.092812 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-credential-socket" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:23.093097 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.093088 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-data" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:23.094192 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.094168 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-kube-api-access-dfwng" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "kube-api-access-dfwng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:48:23.094805 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.094781 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-istio-token" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:48:23.095018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.094995 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-envoy" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:23.095298 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.095276 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7b228db-d145-47dd-8609-76d77564749e-istio-podinfo" pod "f7b228db-d145-47dd-8609-76d77564749e" (UID: "f7b228db-d145-47dd-8609-76d77564749e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:48:23.191945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191903 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwng\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-kube-api-access-dfwng\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.191945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191934 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.191945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191944 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.191945 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191953 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7b228db-d145-47dd-8609-76d77564749e-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.192369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191962 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.192369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191973 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.192369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191981 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7b228db-d145-47dd-8609-76d77564749e-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.192369 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.191990 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7b228db-d145-47dd-8609-76d77564749e-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:23.387973 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.387932 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq"] Oct 31 16:48:23.399166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:23.394333 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m7bsq"] Oct 31 16:48:25.090053 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:25.090003 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b228db-d145-47dd-8609-76d77564749e" path="/var/lib/kubelet/pods/f7b228db-d145-47dd-8609-76d77564749e/volumes" Oct 31 16:48:27.452937 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:27.452902 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:27.593304 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:27.593265 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:30.739507 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:30.739467 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:32.301131 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:32.301089 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:32.587843 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:32.587753 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:32.800226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:32.800190 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5"] Oct 31 16:48:32.800433 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:32.800412 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" containerID="cri-o://36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80" gracePeriod=30 Oct 31 16:48:33.230031 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:33.229994 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:33.746212 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:33.746173 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:33.880510 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:48:33.880475 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:48:33.896872 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:33.896836 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:38.050928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.050882 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:38.052229 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.052209 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:48:38.054894 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.054872 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-certs\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.054964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.054904 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-socket\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.054964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.054952 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istiod-ca-cert\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.055032 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.054981 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-podinfo\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.055066 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055046 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-kube-api-access-cs9v7\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-kube-api-access-cs9v7\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.055107 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055079 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-token\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-token\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.055176 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055111 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-data\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.055176 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055160 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-certs" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:38.055285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055234 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-credential-socket\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.055342 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055286 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-socket" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:38.055402 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055379 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istiod-ca-cert" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:48:38.055484 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055466 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-credential-socket" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:38.055614 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055601 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.055656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055616 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.055656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055625 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.055656 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055633 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.055814 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.055730 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-data" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:38.057263 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.057240 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-podinfo" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:48:38.057728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.057703 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-token" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:48:38.057821 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.057738 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-kube-api-access-cs9v7" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "kube-api-access-cs9v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:48:38.066286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.066254 4880 generic.go:356] "Generic (PLEG): container finished" podID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerID="36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80" exitCode=0 Oct 31 16:48:38.066410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.066302 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" Oct 31 16:48:38.066410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.066323 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" event={"ID":"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b","Type":"ContainerDied","Data":"36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80"} Oct 31 16:48:38.066410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.066386 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5" event={"ID":"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b","Type":"ContainerDied","Data":"a4c70b63b64a6060b769ce077061df22758b86a03418ea6542375c8e8ec11593"} Oct 31 16:48:38.066531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.066418 4880 scope.go:122] "RemoveContainer" containerID="36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80" Oct 31 16:48:38.078815 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.078794 4880 scope.go:122] "RemoveContainer" containerID="36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80" Oct 31 16:48:38.079167 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:48:38.079130 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80\": container with ID starting with 36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80 not found: ID does not exist" containerID="36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80" Oct 31 16:48:38.079257 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.079176 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80"} err="failed to get container status \"36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80\": rpc error: code = NotFound desc = could not find container \"36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80\": container with ID starting with 36f55b549451d93ea69748d4c29a02fc2dc640ed4a71f86699b53db499507e80 not found: ID does not exist" Oct 31 16:48:38.156065 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.156022 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-envoy\") pod \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\" (UID: \"d77566d1-f156-4cdd-b1ca-fe1e8f6c543b\") " Oct 31 16:48:38.156361 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.156346 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.156413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.156364 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs9v7\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-kube-api-access-cs9v7\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.156413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.156374 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.156413 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.156383 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.158075 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.158047 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-envoy" pod "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" (UID: "d77566d1-f156-4cdd-b1ca-fe1e8f6c543b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:38.257085 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.257050 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:38.315964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.315906 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:38.480278 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.480237 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5"] Oct 31 16:48:38.487024 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.486992 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ldrj5"] Oct 31 16:48:38.537468 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:38.537426 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:39.089609 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:39.089525 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" path="/var/lib/kubelet/pods/d77566d1-f156-4cdd-b1ca-fe1e8f6c543b/volumes" Oct 31 16:48:39.411308 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:39.411263 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:39.564280 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:39.564231 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:39.678734 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:39.678641 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl"] Oct 31 16:48:39.678906 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:39.678861 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" containerID="cri-o://765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0" gracePeriod=30 Oct 31 16:48:39.734327 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:39.734285 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:48:42.955747 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:42.955712 4880 patch_prober.go:29] interesting pod/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:48:42.956243 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:42.955787 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:48:44.941916 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:44.941888 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:48:45.020291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020200 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-credential-socket\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020291 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020244 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-certs\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020310 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-token\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-token\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020341 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-kube-api-access-m8299\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-kube-api-access-m8299\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020376 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-envoy\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020427 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-socket\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020458 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-data\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020531 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020491 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-podinfo\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020819 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020566 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istiod-ca-cert\") pod \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\" (UID: \"bf79a2f0-d2ac-45d2-8895-ffd676127f0d\") " Oct 31 16:48:45.020819 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020686 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-credential-socket" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:45.020998 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020975 4880 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-credential-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.020998 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.020989 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-socket" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:45.021177 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.021015 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-certs" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:45.021245 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.021198 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-data" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:45.021319 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.021296 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istiod-ca-cert" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:48:45.022906 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.022880 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-envoy" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:48:45.023018 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.022934 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-token" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:48:45.023076 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.023033 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-kube-api-access-m8299" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "kube-api-access-m8299". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:48:45.023117 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.023096 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-podinfo" pod "bf79a2f0-d2ac-45d2-8895-ffd676127f0d" (UID: "bf79a2f0-d2ac-45d2-8895-ffd676127f0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:48:45.039427 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.039401 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-fsxfh"] Oct 31 16:48:45.039639 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.039616 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" podUID="7daa1454-dc7a-41a9-9b9a-332fe155cb73" containerName="manager" containerID="cri-o://e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827" gracePeriod=30 Oct 31 16:48:45.115835 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.115798 4880 generic.go:356] "Generic (PLEG): container finished" podID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerID="765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0" exitCode=0 Oct 31 16:48:45.116017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.115866 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" Oct 31 16:48:45.116017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.115881 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" event={"ID":"bf79a2f0-d2ac-45d2-8895-ffd676127f0d","Type":"ContainerDied","Data":"765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0"} Oct 31 16:48:45.116017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.115981 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl" event={"ID":"bf79a2f0-d2ac-45d2-8895-ffd676127f0d","Type":"ContainerDied","Data":"92d13b8cd96a10e0841952d6821a4af8d732bfafa4d5b9190c3ae52fdfc5d3b8"} Oct 31 16:48:45.116017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.116010 4880 scope.go:122] "RemoveContainer" containerID="765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0" Oct 31 16:48:45.121651 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121627 4880 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istiod-ca-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.121819 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121796 4880 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.121926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121822 4880 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-token\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.121926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121837 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8299\" (UniqueName: \"kubernetes.io/projected/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-kube-api-access-m8299\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.121926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121852 4880 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-envoy\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.121926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121865 4880 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-workload-socket\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.121926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121878 4880 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-data\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.121926 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.121890 4880 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf79a2f0-d2ac-45d2-8895-ffd676127f0d-istio-podinfo\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.150816 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.150790 4880 scope.go:122] "RemoveContainer" containerID="765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0" Oct 31 16:48:45.151132 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:48:45.151108 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0\": container with ID starting with 765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0 not found: ID does not exist" containerID="765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0" Oct 31 16:48:45.151250 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.151156 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0"} err="failed to get container status \"765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0\": rpc error: code = NotFound desc = could not find container \"765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0\": container with ID starting with 765ed40b36e45889ba23afb9763423fbafa1e8950bce8b362e17516a0d700bb0 not found: ID does not exist" Oct 31 16:48:45.277560 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.277495 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:48:45.305681 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.305649 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl"] Oct 31 16:48:45.311464 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.311439 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f7821a-openshift-default-b4ddf4df7-frjdl"] Oct 31 16:48:45.323145 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.323116 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-metrics-certs\") pod \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " Oct 31 16:48:45.323286 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.323270 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7daa1454-dc7a-41a9-9b9a-332fe155cb73-kube-api-access-dl2xg\" (UniqueName: \"kubernetes.io/projected/7daa1454-dc7a-41a9-9b9a-332fe155cb73-kube-api-access-dl2xg\") pod \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " Oct 31 16:48:45.323354 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.323312 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-webhook-cert\") pod \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\" (UID: \"7daa1454-dc7a-41a9-9b9a-332fe155cb73\") " Oct 31 16:48:45.325277 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.325248 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-metrics-certs" pod "7daa1454-dc7a-41a9-9b9a-332fe155cb73" (UID: "7daa1454-dc7a-41a9-9b9a-332fe155cb73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:48:45.325366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.325291 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7daa1454-dc7a-41a9-9b9a-332fe155cb73-kube-api-access-dl2xg" pod "7daa1454-dc7a-41a9-9b9a-332fe155cb73" (UID: "7daa1454-dc7a-41a9-9b9a-332fe155cb73"). InnerVolumeSpecName "kube-api-access-dl2xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:48:45.325366 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.325308 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-webhook-cert" pod "7daa1454-dc7a-41a9-9b9a-332fe155cb73" (UID: "7daa1454-dc7a-41a9-9b9a-332fe155cb73"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:48:45.424906 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.424868 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2xg\" (UniqueName: \"kubernetes.io/projected/7daa1454-dc7a-41a9-9b9a-332fe155cb73-kube-api-access-dl2xg\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.424906 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.424910 4880 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-webhook-cert\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:45.425130 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:45.424925 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7daa1454-dc7a-41a9-9b9a-332fe155cb73-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 16:48:46.123395 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.123314 4880 generic.go:356] "Generic (PLEG): container finished" podID="7daa1454-dc7a-41a9-9b9a-332fe155cb73" containerID="e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827" exitCode=0 Oct 31 16:48:46.123395 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.123379 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" Oct 31 16:48:46.123903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.123403 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" event={"ID":"7daa1454-dc7a-41a9-9b9a-332fe155cb73","Type":"ContainerDied","Data":"e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827"} Oct 31 16:48:46.123903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.123458 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c95c64d65-fsxfh" event={"ID":"7daa1454-dc7a-41a9-9b9a-332fe155cb73","Type":"ContainerDied","Data":"55d8acb111a8bffb288ebafb518ca7e7b9219de683c1a4dc437339fbe3dfedcc"} Oct 31 16:48:46.123903 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.123479 4880 scope.go:122] "RemoveContainer" containerID="e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827" Oct 31 16:48:46.133923 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.133906 4880 scope.go:122] "RemoveContainer" containerID="e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827" Oct 31 16:48:46.134210 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:48:46.134189 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827\": container with ID starting with e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827 not found: ID does not exist" containerID="e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827" Oct 31 16:48:46.134259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.134219 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827"} err="failed to get container status \"e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827\": rpc error: code = NotFound desc = could not find container \"e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827\": container with ID starting with e0d0bf6ba0fbe78a36b807aae08a7fb0c68d6f98025eb9b709a1b3bb1652e827 not found: ID does not exist" Oct 31 16:48:46.275829 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.275793 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-fsxfh"] Oct 31 16:48:46.322198 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:46.322165 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c95c64d65-fsxfh"] Oct 31 16:48:47.090889 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:47.090862 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7daa1454-dc7a-41a9-9b9a-332fe155cb73" path="/var/lib/kubelet/pods/7daa1454-dc7a-41a9-9b9a-332fe155cb73/volumes" Oct 31 16:48:47.091351 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:48:47.091336 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" path="/var/lib/kubelet/pods/bf79a2f0-d2ac-45d2-8895-ffd676127f0d/volumes" Oct 31 16:49:03.881094 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:49:03.881053 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:49:05.054022 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:05.053986 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:05.252160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:05.252102 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:22.024612 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:22.024573 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:22.215779 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:22.215743 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:22.963564 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:22.963519 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:25.405622 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:25.405590 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:33.881946 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:49:33.881901 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:49:38.878776 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:38.878737 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:41.484299 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:41.484262 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:55.734233 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:55.734198 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:49:56.003226 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:49:56.003122 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:50:03.881526 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:50:03.881485 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:50:30.381798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:30.381761 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:50:30.533808 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:30.533771 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:50:33.703259 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703229 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr"] Oct 31 16:50:33.703632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703598 4880 cpu_manager.go:414] "containerMap: removing container" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" Oct 31 16:50:33.703632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703609 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" Oct 31 16:50:33.703632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703624 4880 cpu_manager.go:414] "containerMap: removing container" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.703632 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703630 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703642 4880 cpu_manager.go:414] "containerMap: removing container" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703647 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703655 4880 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703659 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703666 4880 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703672 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703679 4880 cpu_manager.go:414] "containerMap: removing container" podUID="7daa1454-dc7a-41a9-9b9a-332fe155cb73" containerName="manager" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703683 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7daa1454-dc7a-41a9-9b9a-332fe155cb73" containerName="manager" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703689 4880 cpu_manager.go:414] "containerMap: removing container" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703695 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703701 4880 cpu_manager.go:414] "containerMap: removing container" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703705 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703712 4880 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703716 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703730 4880 cpu_manager.go:414] "containerMap: removing container" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703734 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703750 4880 cpu_manager.go:414] "containerMap: removing container" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" Oct 31 16:50:33.703762 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703754 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703811 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77566d1-f156-4cdd-b1ca-fe1e8f6c543b" containerName="istio-proxy" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703819 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703827 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b228db-d145-47dd-8609-76d77564749e" containerName="istio-proxy" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703834 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd4f876b-52f7-41af-bd1f-63ff7ccc42be" containerName="main" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703840 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703846 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c36c9a-01b5-42eb-9ae9-23717f85bd8c" containerName="main" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703851 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703857 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703862 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="7daa1454-dc7a-41a9-9b9a-332fe155cb73" containerName="manager" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703868 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf79a2f0-d2ac-45d2-8895-ffd676127f0d" containerName="istio-proxy" Oct 31 16:50:33.704309 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.703873 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2e60ba-2d45-4fa0-ae0f-1ab5ab719d34" containerName="cleanup" Oct 31 16:50:33.707290 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.707273 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.749148 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.749107 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb"] Oct 31 16:50:33.749593 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.749578 4880 cpu_manager.go:414] "containerMap: removing container" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.749641 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.749595 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.749700 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.749690 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ede67c-6fd3-4635-8612-8e98b18a89a0" containerName="cleanup" Oct 31 16:50:33.752977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.752955 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.800802 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.800767 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr"] Oct 31 16:50:33.803601 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803576 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.803689 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803611 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.803689 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803649 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.803768 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803703 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.803768 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803746 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.803845 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803783 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbmg\" (UniqueName: \"kubernetes.io/projected/edd28810-e327-41db-9a4f-156145eb97d5-kube-api-access-jdbmg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.803889 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803853 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.803889 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803873 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd28810-e327-41db-9a4f-156145eb97d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.803964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803892 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd28810-e327-41db-9a4f-156145eb97d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.803964 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803909 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.804039 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.803975 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.804124 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.804098 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.804215 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.804200 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.804252 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.804236 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd28810-e327-41db-9a4f-156145eb97d5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.804295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.804256 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.804295 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.804284 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.804410 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.804392 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.804453 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.804429 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbv5\" (UniqueName: \"kubernetes.io/projected/31e2f962-1460-46a6-9a3c-f44a20b69a2c-kube-api-access-flbv5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.853798 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.853748 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb"] Oct 31 16:50:33.881537 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:50:33.881500 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:50:33.905207 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905180 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905224 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905267 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905293 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbv5\" (UniqueName: \"kubernetes.io/projected/31e2f962-1460-46a6-9a3c-f44a20b69a2c-kube-api-access-flbv5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905313 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905365 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905338 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905521 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905570 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905594 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905620 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbmg\" (UniqueName: \"kubernetes.io/projected/edd28810-e327-41db-9a4f-156145eb97d5-kube-api-access-jdbmg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905648 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905676 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905705 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd28810-e327-41db-9a4f-156145eb97d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905737 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd28810-e327-41db-9a4f-156145eb97d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905745 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905772 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905805 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.905862 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905838 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.906339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905877 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.906339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905885 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.906339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905933 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd28810-e327-41db-9a4f-156145eb97d5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.906339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.906164 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.906339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.905701 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.906339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.906267 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.906938 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.906915 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.907067 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.907048 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd28810-e327-41db-9a4f-156145eb97d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.907364 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.907327 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.907482 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.907456 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.908009 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.907978 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.908403 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.908378 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd28810-e327-41db-9a4f-156145eb97d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.908489 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.908432 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd28810-e327-41db-9a4f-156145eb97d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.908537 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.908513 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.959190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.959074 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e2f962-1460-46a6-9a3c-f44a20b69a2c-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:33.959190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.959105 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd28810-e327-41db-9a4f-156145eb97d5-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.959190 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.959115 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbmg\" (UniqueName: \"kubernetes.io/projected/edd28810-e327-41db-9a4f-156145eb97d5-kube-api-access-jdbmg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-sr5pb\" (UID: \"edd28810-e327-41db-9a4f-156145eb97d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:33.959441 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:33.959414 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbv5\" (UniqueName: \"kubernetes.io/projected/31e2f962-1460-46a6-9a3c-f44a20b69a2c-kube-api-access-flbv5\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m4gjr\" (UID: \"31e2f962-1460-46a6-9a3c-f44a20b69a2c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:34.020516 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.020480 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:34.066203 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.066170 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:34.443657 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.443621 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr"] Oct 31 16:50:34.492535 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:50:34.492500 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e2f962_1460_46a6_9a3c_f44a20b69a2c.slice/crio-3d8b7d3f40ecc7f9332bbe773a613e43cc874a84fad62b03f9f075dea0d1a3b4 WatchSource:0}: Error finding container 3d8b7d3f40ecc7f9332bbe773a613e43cc874a84fad62b03f9f075dea0d1a3b4: Status 404 returned error can't find the container with id 3d8b7d3f40ecc7f9332bbe773a613e43cc874a84fad62b03f9f075dea0d1a3b4 Oct 31 16:50:34.494737 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.494706 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:50:34.494826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.494808 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:50:34.494826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.494839 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:50:34.498002 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.497980 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb"] Oct 31 16:50:34.504933 ip-10-0-142-127 kubenswrapper[4880]: W1031 16:50:34.504913 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd28810_e327_41db_9a4f_156145eb97d5.slice/crio-fa874b5f8a2f60e09f6dbc5e046871d27f8dbe5f3de6ad3c409f48ac1beb93ee WatchSource:0}: Error finding container fa874b5f8a2f60e09f6dbc5e046871d27f8dbe5f3de6ad3c409f48ac1beb93ee: Status 404 returned error can't find the container with id fa874b5f8a2f60e09f6dbc5e046871d27f8dbe5f3de6ad3c409f48ac1beb93ee Oct 31 16:50:34.507216 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.507173 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:50:34.507297 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.507262 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:50:34.507337 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.507307 4880 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 16:50:34.726258 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.726129 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" event={"ID":"edd28810-e327-41db-9a4f-156145eb97d5","Type":"ContainerStarted","Data":"3936c186c4f5e6664117420305de1b5cf705bc06709ff44968484f25279065cc"} Oct 31 16:50:34.726258 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.726214 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" event={"ID":"edd28810-e327-41db-9a4f-156145eb97d5","Type":"ContainerStarted","Data":"fa874b5f8a2f60e09f6dbc5e046871d27f8dbe5f3de6ad3c409f48ac1beb93ee"} Oct 31 16:50:34.734184 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.728609 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" event={"ID":"31e2f962-1460-46a6-9a3c-f44a20b69a2c","Type":"ContainerStarted","Data":"c9b1e25cd6678d975d13c623d40964f4aa45642b2e5f12aaebde88c2526fac20"} Oct 31 16:50:34.734184 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.728673 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" event={"ID":"31e2f962-1460-46a6-9a3c-f44a20b69a2c","Type":"ContainerStarted","Data":"3d8b7d3f40ecc7f9332bbe773a613e43cc874a84fad62b03f9f075dea0d1a3b4"} Oct 31 16:50:34.838524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.838459 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podStartSLOduration=1.838440882 podStartE2EDuration="1.838440882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:50:34.835285481 +0000 UTC m=+2881.870580345" watchObservedRunningTime="2026-10-31 16:50:34.838440882 +0000 UTC m=+2881.873735689" Oct 31 16:50:34.947841 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:34.947776 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podStartSLOduration=1.947755188 podStartE2EDuration="1.947755188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:50:34.94355615 +0000 UTC m=+2881.978850956" watchObservedRunningTime="2026-10-31 16:50:34.947755188 +0000 UTC m=+2881.983050009" Oct 31 16:50:35.021228 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:35.021076 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:35.025704 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:35.025268 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:35.025704 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:35.025343 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:35.072160 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:35.070762 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:35.076165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:35.075691 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:35.076165 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:35.075751 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:36.028166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:36.024546 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:36.028166 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:36.024610 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:36.068164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:36.067661 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:36.068164 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:36.067735 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:37.024591 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:37.024543 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:37.024884 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:37.024851 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:37.070646 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:37.070607 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:37.071272 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:37.071231 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:38.024583 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:38.024542 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:38.024867 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:38.024837 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:38.069806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:38.069624 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:38.069806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:38.069691 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:39.058161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:39.045533 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:39.058161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:39.045600 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:39.070595 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:39.070548 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:39.070922 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:39.070891 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:39.280014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:39.279981 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:50:39.415819 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:39.415793 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:50:40.028161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:40.024488 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:40.028161 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:40.024543 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:40.069727 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:40.069683 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:40.070247 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:40.069761 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:41.021204 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:41.021171 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:41.021392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:41.021227 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:41.067487 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:41.067448 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:41.067636 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:41.067512 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:42.022156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:42.021710 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 16:50:42.022156 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:42.021781 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 16:50:42.066871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:42.066727 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 31 16:50:42.066871 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:42.066796 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 31 16:50:44.022004 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:44.021958 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:50:44.022551 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:44.022044 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:50:44.068098 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:44.068057 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:50:44.068326 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:44.068120 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:50:45.064439 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.064371 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:50:45.065065 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.065028 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:50:45.109358 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.109321 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:50:45.109516 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.109401 4880 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:50:45.289017 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.288978 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:45.289561 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.289519 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:45.647421 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.647392 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" Oct 31 16:50:45.769819 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.769789 4880 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:45.770430 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.770408 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:50:45.771220 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:50:45.771200 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" Oct 31 16:51:03.881273 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:51:03.881232 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:51:33.887683 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:51:33.887644 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:51:39.573269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:51:39.573230 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:52:03.880174 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:52:03.880112 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:52:33.595772 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:52:33.595742 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:52:33.604050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:52:33.604024 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:52:33.606830 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:52:33.606803 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:52:33.614371 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:52:33.614349 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:52:33.614497 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:52:33.614481 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:52:33.625163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:52:33.625130 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:52:33.884643 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:52:33.884608 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:52:39.914996 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:52:39.914960 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:53:03.879910 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:53:03.879869 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:53:13.467675 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:53:13.467632 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:53:15.866718 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:53:15.866666 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:53:33.882434 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:53:33.882397 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:53:40.265416 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:53:40.265372 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:54:03.915478 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:54:03.915437 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:54:16.217832 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:54:16.217745 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:54:33.880034 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:54:33.879994 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:54:40.628849 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:54:40.628809 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:55:03.880561 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:55:03.880521 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:55:16.818328 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:55:16.818272 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m4gjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:55:16.818757 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:55:16.818357 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m4gjr" podUID="31e2f962-1460-46a6-9a3c-f44a20b69a2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:55:33.881691 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:55:33.881654 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:55:40.901305 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:55:40.901272 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:55:46.948877 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:55:46.948790 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:03.881940 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:56:03.881896 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:56:29.866604 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:29.866567 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:30.020755 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:30.020698 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:33.699855 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:33.699822 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:33.881626 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:56:33.881589 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:56:33.894782 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:33.894740 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:34.048069 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:34.047990 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:36.926535 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:36.926501 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:37.126765 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:37.126727 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:37.902422 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:37.902391 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:40.746312 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:40.746271 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:40.933553 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:40.933516 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:41.236836 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:41.236800 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:46.102647 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:46.102609 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:51.342033 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:51.341995 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:56.519991 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:56.519952 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:58.559918 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:58.559880 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:59.089620 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:59.089587 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:59.256392 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:59.256353 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:56:59.415958 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:56:59.415916 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:01.892977 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:01.892938 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:02.137728 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:02.137693 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:03.882605 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:57:03.882571 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:57:07.320014 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:07.319974 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:12.603599 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:12.603555 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:17.804806 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:17.804772 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:23.012348 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:23.012307 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:28.192813 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:28.192772 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:33.449414 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:33.449372 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:33.638577 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:33.638546 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:57:33.646928 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:33.646899 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:57:33.650840 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:33.650816 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 16:57:33.658364 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:33.658338 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:57:33.658652 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:33.658638 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 16:57:33.669285 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:33.669267 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 16:57:33.880624 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:57:33.880536 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:57:38.619264 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:38.619220 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:41.569826 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:41.569787 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:43.772339 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:43.772306 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:48.923269 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:48.923226 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:54.113620 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:54.113578 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:58.623851 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:58.623811 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:57:59.297163 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:57:59.297089 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:03.658504 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:03.658464 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:03.881760 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:58:03.881721 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:58:05.077616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:05.077553 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:10.210050 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:10.210014 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:10.957631 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:10.957597 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:11.145616 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:11.145576 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:14.815558 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:14.815510 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:16.470294 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:16.470263 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:16.774492 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:16.774408 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:17.532580 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:17.532544 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:23.140314 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:23.140274 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:23.334524 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:23.334480 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:23.513913 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:23.513827 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:33.880722 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:58:33.880684 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:58:39.848702 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:39.848664 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:40.505360 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:40.505323 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:41.967255 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:41.967212 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:58:46.091758 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:58:46.091669 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 16:59:03.881804 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:59:03.881764 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:59:33.880411 ip-10-0-142-127 kubenswrapper[4880]: E1031 16:59:33.880373 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 16:59:42.344939 ip-10-0-142-127 kubenswrapper[4880]: I1031 16:59:42.344902 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:00:03.879933 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:00:03.879896 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:00:33.879860 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:00:33.879821 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:00:42.642426 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:00:42.642379 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:03.879866 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:01:03.879824 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:01:10.839508 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:10.839468 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:11.506078 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:11.506039 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:11.736287 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:11.736251 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:16.205095 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:16.205060 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:16.385405 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:16.385377 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:16.542202 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:16.542101 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:18.716351 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:18.716312 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:19.372006 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:19.371970 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:19.527963 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:19.527922 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:23.246607 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:23.246572 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:23.416112 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:23.416082 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:23.564862 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:23.564778 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:28.600266 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:28.600226 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:33.849795 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:33.849759 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:33.879758 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:01:33.879723 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:01:39.107468 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:39.107434 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:41.001241 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:41.001201 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:41.584714 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:41.584683 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:41.751591 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:41.751552 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:41.907334 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:41.907291 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:43.010745 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:43.010672 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:44.754276 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:44.754238 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:45.157775 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:45.157743 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:47.774201 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:47.774163 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:49.915068 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:49.915038 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:55.101118 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:55.101079 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:57.455426 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:57.455387 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:01:57.643452 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:01:57.643416 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:00.378469 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:00.378435 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:03.881006 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:02:03.880957 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:02:05.509403 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:05.509364 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:10.630330 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:10.630285 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:15.820002 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:15.819962 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:20.934575 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:20.934541 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:26.202992 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:26.202955 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:31.314173 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:31.314119 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:33.682153 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:33.682110 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 17:02:33.690493 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:33.690465 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 17:02:33.698220 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:33.698195 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 17:02:33.705012 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:33.704985 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 17:02:33.708196 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:33.708175 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 17:02:33.718614 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:33.718594 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 17:02:33.880876 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:02:33.880840 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:02:36.524197 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:36.524158 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:41.076791 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:41.076745 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:41.798933 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:41.798897 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:43.296885 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:43.296856 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:46.163050 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:46.163010 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:47.551477 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:47.551440 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:52.722084 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:52.722053 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:53.411026 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:53.410983 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:53.558125 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:53.558080 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:57.269526 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:57.269487 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:58.906734 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:58.906700 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:59.202534 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:59.202450 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:02:59.985936 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:02:59.985897 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:03.880525 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:03:03.880486 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:03:05.832902 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:05.832869 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:05.993395 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:05.993356 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:06.174427 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:06.174386 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:13.669104 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:13.669051 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:22.785033 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:22.784989 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:23.186915 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:23.186872 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:33.880620 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:03:33.880574 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:03:37.134363 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:37.134329 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:43.629607 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:43.629568 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:03:53.847265 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:03:53.847225 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:04:01.291042 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:04:01.290960 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-sr5pb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:04:01.291042 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:04:01.291030 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-sr5pb" podUID="edd28810-e327-41db-9a4f-156145eb97d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:04:03.880111 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:04:03.880073 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:04:33.880764 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:04:33.880722 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:04:37.606406 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:04:37.606363 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:04:42.157648 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:04:42.157589 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:04:44.049907 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:04:44.049794 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:04:51.312047 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:04:51.312013 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:03.881259 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:05:03.881212 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:05:07.964660 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:07.964242 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:05:07.964660 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:07.964329 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:05:33.881851 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:05:33.881808 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:05:38.737418 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:38.737377 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:39.351693 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:39.351657 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:39.559076 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:39.559038 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:40.826467 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:40.826435 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj"] Oct 31 17:05:40.830254 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:40.830229 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:40.928360 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:40.928326 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj"] Oct 31 17:05:40.997982 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:40.997934 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f62a696-eb18-4347-9be4-f75cad494736-openshift-service-ca\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:40.998225 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:40.997998 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f62a696-eb18-4347-9be4-f75cad494736-metrics-certs\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:40.998225 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:40.998123 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9ksj\" (UniqueName: \"kubernetes.io/projected/2f62a696-eb18-4347-9be4-f75cad494736-kube-api-access-s9ksj\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:40.998225 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:40.998198 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f62a696-eb18-4347-9be4-f75cad494736-maas-api-tls\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.098981 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.098897 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f62a696-eb18-4347-9be4-f75cad494736-openshift-service-ca\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.098981 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.098934 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f62a696-eb18-4347-9be4-f75cad494736-metrics-certs\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.099261 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.098993 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9ksj\" (UniqueName: \"kubernetes.io/projected/2f62a696-eb18-4347-9be4-f75cad494736-kube-api-access-s9ksj\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.099261 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.099020 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f62a696-eb18-4347-9be4-f75cad494736-maas-api-tls\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.099694 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.099666 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f62a696-eb18-4347-9be4-f75cad494736-openshift-service-ca\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.101460 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.101432 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f62a696-eb18-4347-9be4-f75cad494736-maas-api-tls\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.101570 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.101515 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f62a696-eb18-4347-9be4-f75cad494736-metrics-certs\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.151344 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.151311 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9ksj\" (UniqueName: \"kubernetes.io/projected/2f62a696-eb18-4347-9be4-f75cad494736-kube-api-access-s9ksj\") pod \"maas-api-67bf6fc4b5-ns5bj\" (UID: \"2f62a696-eb18-4347-9be4-f75cad494736\") " pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.442610 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.442580 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:41.813596 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.813517 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj"] Oct 31 17:05:41.819741 ip-10-0-142-127 kubenswrapper[4880]: W1031 17:05:41.819712 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f62a696_eb18_4347_9be4_f75cad494736.slice/crio-3f12d084d4ba8812b5443f10d65c18aeb05788c41eb7d9ed4c4e7cd54be2da87 WatchSource:0}: Error finding container 3f12d084d4ba8812b5443f10d65c18aeb05788c41eb7d9ed4c4e7cd54be2da87: Status 404 returned error can't find the container with id 3f12d084d4ba8812b5443f10d65c18aeb05788c41eb7d9ed4c4e7cd54be2da87 Oct 31 17:05:41.821065 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:41.821047 4880 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 17:05:42.393776 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:42.393738 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" event={"ID":"2f62a696-eb18-4347-9be4-f75cad494736","Type":"ContainerStarted","Data":"3f12d084d4ba8812b5443f10d65c18aeb05788c41eb7d9ed4c4e7cd54be2da87"} Oct 31 17:05:43.139631 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:43.139597 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:43.365464 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:43.365431 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:43.401019 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:43.400932 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" event={"ID":"2f62a696-eb18-4347-9be4-f75cad494736","Type":"ContainerStarted","Data":"b7ae762c113125cdc0c311da778a6b0d81a31b8723fbc6432560738fed1fc0ef"} Oct 31 17:05:43.401408 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:43.401059 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:43.512243 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:43.512192 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" podStartSLOduration=2.6290678119999997 podStartE2EDuration="3.512178501s" totalImagesPullingTime="883.110689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 17:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:05:43.511109051 +0000 UTC m=+3790.546403856" watchObservedRunningTime="2026-10-31 17:05:43.512178501 +0000 UTC m=+3790.547473305" Oct 31 17:05:44.387582 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:44.387553 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:46.253061 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:46.253017 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:46.451605 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:46.451565 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:49.056949 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:49.056909 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:49.236844 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:49.236810 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:49.392915 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:49.392883 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:05:49.411580 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:49.411546 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67bf6fc4b5-ns5bj" Oct 31 17:05:49.619173 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:49.619114 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9"] Oct 31 17:05:49.619602 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:49.619562 4880 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" podUID="6011e6c0-bc37-4a57-b855-1a2b8a98670b" containerName="maas-api" containerID="cri-o://aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e" gracePeriod=30 Oct 31 17:05:50.861302 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.861278 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 17:05:50.892607 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.892579 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-maas-api-tls\") pod \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " Oct 31 17:05:50.892744 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.892643 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6011e6c0-bc37-4a57-b855-1a2b8a98670b-kube-api-access-8m46f\" (UniqueName: \"kubernetes.io/projected/6011e6c0-bc37-4a57-b855-1a2b8a98670b-kube-api-access-8m46f\") pod \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " Oct 31 17:05:50.892744 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.892685 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-metrics-certs\") pod \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " Oct 31 17:05:50.892744 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.892726 4880 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6011e6c0-bc37-4a57-b855-1a2b8a98670b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6011e6c0-bc37-4a57-b855-1a2b8a98670b-openshift-service-ca\") pod \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\" (UID: \"6011e6c0-bc37-4a57-b855-1a2b8a98670b\") " Oct 31 17:05:50.893115 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.893086 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6011e6c0-bc37-4a57-b855-1a2b8a98670b-openshift-service-ca" pod "6011e6c0-bc37-4a57-b855-1a2b8a98670b" (UID: "6011e6c0-bc37-4a57-b855-1a2b8a98670b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 17:05:50.894627 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.894602 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-maas-api-tls" pod "6011e6c0-bc37-4a57-b855-1a2b8a98670b" (UID: "6011e6c0-bc37-4a57-b855-1a2b8a98670b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:05:50.894792 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.894773 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-metrics-certs" pod "6011e6c0-bc37-4a57-b855-1a2b8a98670b" (UID: "6011e6c0-bc37-4a57-b855-1a2b8a98670b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 17:05:50.894857 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.894775 4880 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6011e6c0-bc37-4a57-b855-1a2b8a98670b-kube-api-access-8m46f" pod "6011e6c0-bc37-4a57-b855-1a2b8a98670b" (UID: "6011e6c0-bc37-4a57-b855-1a2b8a98670b"). InnerVolumeSpecName "kube-api-access-8m46f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 17:05:50.993614 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.993542 4880 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-metrics-certs\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 17:05:50.993614 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.993573 4880 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6011e6c0-bc37-4a57-b855-1a2b8a98670b-openshift-service-ca\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 17:05:50.993614 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.993589 4880 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6011e6c0-bc37-4a57-b855-1a2b8a98670b-maas-api-tls\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 17:05:50.993614 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:50.993604 4880 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m46f\" (UniqueName: \"kubernetes.io/projected/6011e6c0-bc37-4a57-b855-1a2b8a98670b-kube-api-access-8m46f\") on node \"ip-10-0-142-127.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 17:05:51.454096 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.454061 4880 generic.go:356] "Generic (PLEG): container finished" podID="6011e6c0-bc37-4a57-b855-1a2b8a98670b" containerID="aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e" exitCode=0 Oct 31 17:05:51.454321 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.454112 4880 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" Oct 31 17:05:51.454321 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.454115 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" event={"ID":"6011e6c0-bc37-4a57-b855-1a2b8a98670b","Type":"ContainerDied","Data":"aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e"} Oct 31 17:05:51.454321 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.454246 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9" event={"ID":"6011e6c0-bc37-4a57-b855-1a2b8a98670b","Type":"ContainerDied","Data":"2656d3e8cf88d7aaaae1f8480724b48e7e08f018d7640d0a9d36baafa2aec62c"} Oct 31 17:05:51.454321 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.454268 4880 scope.go:122] "RemoveContainer" containerID="aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e" Oct 31 17:05:51.463954 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.463932 4880 scope.go:122] "RemoveContainer" containerID="aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e" Oct 31 17:05:51.464280 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:05:51.464261 4880 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e\": container with ID starting with aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e not found: ID does not exist" containerID="aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e" Oct 31 17:05:51.464335 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.464289 4880 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e"} err="failed to get container status \"aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e\": rpc error: code = NotFound desc = could not find container \"aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e\": container with ID starting with aebf01ebcc91b45e5e3dc7a187953f55752d323077b2d66df8757223f9204d3e not found: ID does not exist" Oct 31 17:05:51.605612 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.605576 4880 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9"] Oct 31 17:05:51.611320 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:51.611293 4880 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bfdbbc5d5-l6xq9"] Oct 31 17:05:53.090324 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:53.090291 4880 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6011e6c0-bc37-4a57-b855-1a2b8a98670b" path="/var/lib/kubelet/pods/6011e6c0-bc37-4a57-b855-1a2b8a98670b/volumes" Oct 31 17:05:54.816080 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:05:54.816043 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:00.069204 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:00.069161 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:03.882625 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:06:03.882591 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:06:05.298855 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:05.298814 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:11.213928 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:11.213890 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:16.613580 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:16.613484 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:21.803314 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:21.803272 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:26.950728 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:26.950681 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:31.198292 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:31.198234 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:31.390122 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:31.390082 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:31.532682 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:31.532585 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:33.881489 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:06:33.881453 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:06:35.144082 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:35.144037 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:35.461094 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:35.460997 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:40.435739 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:40.435688 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:41.564532 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:41.564489 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:42.161825 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:42.161786 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:42.366950 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:42.366912 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:44.761372 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:44.761330 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:45.961236 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:45.961202 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:46.141263 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:46.141229 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:46.302020 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:46.301922 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:49.156749 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:49.156718 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:49.350410 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:49.350363 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:53.007871 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:53.007834 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:53.195778 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:53.195745 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:06:58.332256 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:06:58.332218 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:03.640090 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:03.640049 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:03.881889 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:07:03.881852 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:07:04.609699 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:04.609660 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:04.765532 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:04.765493 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:08.816776 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:08.816730 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:10.672092 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:10.672047 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:14.556089 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:14.556046 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:19.464403 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:19.464345 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:19.701020 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:19.700988 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:24.914533 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:24.914496 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:30.025767 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:30.025729 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:33.727084 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:33.727057 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 17:07:33.734945 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:33.734924 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 17:07:33.743825 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:33.743801 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 17:07:33.746592 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:33.746572 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 17:07:33.751719 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:33.751702 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 17:07:33.762379 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:33.762363 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 17:07:33.880744 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:07:33.880707 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:07:34.696829 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:34.696783 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:35.204389 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:35.204353 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:40.315304 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:40.315261 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:45.063165 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:45.063025 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:47.695745 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:47.695706 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:50.627745 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:50.627710 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:07:55.813061 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:07:55.813004 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:00.917831 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:00.917788 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:03.880711 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:08:03.880674 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:08:06.121438 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:06.121402 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:11.320003 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:11.319970 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:16.504095 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:16.504050 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:21.703651 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:21.703606 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:26.950423 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:26.950378 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:32.199241 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:32.199201 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:33.885623 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:08:33.885585 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:08:37.808906 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:37.808866 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:42.711587 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:42.711551 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:42.858695 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:42.858654 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:43.011364 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:43.011265 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:45.386401 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:45.386361 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:52.689408 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:52.689368 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:54.325235 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:54.325195 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:54.637298 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:54.637257 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:08:55.376769 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:08:55.376734 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:01.042063 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:01.042026 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:01.236747 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:01.236709 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:01.392984 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:01.392936 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:01.679375 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:01.679263 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:03.879926 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:09:03.879885 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:09:22.963834 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:22.963682 4880 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-jdhp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 17:09:22.964678 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:22.964642 4880 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-jdhp6" podUID="363aa6f3-aaba-4371-b953-557fc18a8d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 17:09:33.880318 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:09:33.880276 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:09:39.447353 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:39.447318 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:39.587317 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:39.587281 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:45.709903 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:45.709867 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:53.021401 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:53.021364 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:09:53.210354 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:09:53.210279 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:10:03.879209 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:10:03.879164 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:10:12.989393 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:12.989355 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6796fd4df4-27p22_61704b91-19e2-4915-9f4f-2a1dbf0307c6/authorino/0.log" Oct 31 17:10:24.237603 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:24.237567 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9l8zz_27b75c60-a73e-4281-ace2-d2fa046c2f6d/manager/0.log" Oct 31 17:10:24.749796 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:24.749763 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s9485_c064c108-1022-402c-8c87-761962ef7747/manager/0.log" Oct 31 17:10:26.076184 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:26.076128 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ld2rc_e2b1a4b7-4b50-45ae-a68e-af611a888a54/manager/0.log" Oct 31 17:10:26.309206 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:26.309177 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6qjm9_226b0933-6b0f-471a-a4cf-be67d2fbed67/proxy/0.log" Oct 31 17:10:27.396423 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:27.396387 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67bf6fc4b5-ns5bj_2f62a696-eb18-4347-9be4-f75cad494736/maas-api/0.log" Oct 31 17:10:28.329063 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:28.329033 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-f7jt6_ef573716-f984-4646-855d-60a8d4315097/postgres/0.log" Oct 31 17:10:29.832568 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:29.832535 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6796fd4df4-27p22_61704b91-19e2-4915-9f4f-2a1dbf0307c6/authorino/0.log" Oct 31 17:10:30.292471 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:30.292435 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qmlh_cb9fb5ff-4889-4cf7-aa3a-37a80b636932/manager/1.log" Oct 31 17:10:31.775746 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:31.775712 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f5vdt_ecf4590e-572d-4c90-ad77-e130e762b449/manager/0.log" Oct 31 17:10:32.290642 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:32.290600 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qmlh_cb9fb5ff-4889-4cf7-aa3a-37a80b636932/manager/0.log" Oct 31 17:10:32.863339 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:32.863306 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cjljn_b122fe05-653c-4619-a231-c72eefa985b4/istio-proxy/0.log" Oct 31 17:10:33.879901 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:10:33.879867 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:10:34.767840 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:34.767805 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-jdhp6_363aa6f3-aaba-4371-b953-557fc18a8d3f/istio-proxy/1.log" Oct 31 17:10:35.046380 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:35.046301 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-m4gjr_31e2f962-1460-46a6-9a3c-f44a20b69a2c/istio-proxy/0.log" Oct 31 17:10:35.711096 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:35.711065 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-sr5pb_edd28810-e327-41db-9a4f-156145eb97d5/istio-proxy/0.log" Oct 31 17:10:36.206743 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:36.206711 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl_8a5711ec-01b8-47ac-9349-d8d1184711b3/istio-proxy/0.log" Oct 31 17:10:36.872647 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:36.872618 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746fc4dd59-qfxr9_efec2af7-bad8-43d3-8fd8-45071136bdcf/router/0.log" Oct 31 17:10:37.386326 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:37.386293 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-jdhp6_363aa6f3-aaba-4371-b953-557fc18a8d3f/istio-proxy/0.log" Oct 31 17:10:40.656775 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:40.656748 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6796fd4df4-27p22_61704b91-19e2-4915-9f4f-2a1dbf0307c6/authorino/0.log" Oct 31 17:10:46.058560 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:46.058478 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:10:50.209420 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:50.209389 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9l8zz_27b75c60-a73e-4281-ace2-d2fa046c2f6d/manager/0.log" Oct 31 17:10:50.732586 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:50.732552 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s9485_c064c108-1022-402c-8c87-761962ef7747/manager/0.log" Oct 31 17:10:51.662363 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:51.662329 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ld2rc_e2b1a4b7-4b50-45ae-a68e-af611a888a54/manager/0.log" Oct 31 17:10:51.897414 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:51.897376 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6qjm9_226b0933-6b0f-471a-a4cf-be67d2fbed67/proxy/0.log" Oct 31 17:10:52.768052 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:52.768020 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67bf6fc4b5-ns5bj_2f62a696-eb18-4347-9be4-f75cad494736/maas-api/0.log" Oct 31 17:10:53.721352 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:53.721318 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-f7jt6_ef573716-f984-4646-855d-60a8d4315097/postgres/0.log" Oct 31 17:10:54.896452 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:54.896421 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6796fd4df4-27p22_61704b91-19e2-4915-9f4f-2a1dbf0307c6/authorino/0.log" Oct 31 17:10:55.368380 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:55.368290 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qmlh_cb9fb5ff-4889-4cf7-aa3a-37a80b636932/manager/1.log" Oct 31 17:10:56.632797 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:56.632761 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f5vdt_ecf4590e-572d-4c90-ad77-e130e762b449/manager/0.log" Oct 31 17:10:57.311100 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:57.311068 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cjljn_b122fe05-653c-4619-a231-c72eefa985b4/istio-proxy/0.log" Oct 31 17:10:59.273819 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:59.273787 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-jdhp6_363aa6f3-aaba-4371-b953-557fc18a8d3f/istio-proxy/1.log" Oct 31 17:10:59.553224 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:10:59.553105 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-m4gjr_31e2f962-1460-46a6-9a3c-f44a20b69a2c/istio-proxy/0.log" Oct 31 17:11:00.267193 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:11:00.267165 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-sr5pb_edd28810-e327-41db-9a4f-156145eb97d5/istio-proxy/0.log" Oct 31 17:11:00.810552 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:11:00.810519 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-s4mjl_8a5711ec-01b8-47ac-9349-d8d1184711b3/istio-proxy/0.log" Oct 31 17:11:01.470151 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:11:01.470110 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-746fc4dd59-qfxr9_efec2af7-bad8-43d3-8fd8-45071136bdcf/router/0.log" Oct 31 17:11:03.879939 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:11:03.879897 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:11:33.881475 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:11:33.881436 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:11:46.396747 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:11:46.396713 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:12:03.880627 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:12:03.880589 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:12:10.400950 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:10.400900 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:12:10.607151 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:10.607101 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:12:20.051886 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:20.051804 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mvztk_5a66684b-1ce2-4892-b466-d95589fcf351/global-pull-secret-syncer/0.log" Oct 31 17:12:20.599123 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:20.599093 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-69955_45d11ee0-d36c-4f4a-915c-fcedef344e68/connection-checker/0.log" Oct 31 17:12:21.001595 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:21.001568 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9cdsm_ed820d3a-f16c-4bf0-a30e-85ad2732ab58/connection-checker/0.log" Oct 31 17:12:21.403057 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:21.403031 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-x72jc_b3d75aa1-14ee-4dd6-8216-440abcd707fa/connection-checker/0.log" Oct 31 17:12:21.859477 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:21.859386 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" Oct 31 17:12:23.974611 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:23.974574 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-127.us-west-2.compute.internal_10be822f2f376ae4d26eb7bb3d2aeb64/haproxy/0.log" Oct 31 17:12:29.253395 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.253345 4880 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s"] Oct 31 17:12:29.254017 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.253994 4880 cpu_manager.go:414] "containerMap: removing container" podUID="6011e6c0-bc37-4a57-b855-1a2b8a98670b" containerName="maas-api" Oct 31 17:12:29.254086 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.254021 4880 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6011e6c0-bc37-4a57-b855-1a2b8a98670b" containerName="maas-api" Oct 31 17:12:29.254164 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.254148 4880 memory_manager.go:365] "RemoveStaleState removing state" podUID="6011e6c0-bc37-4a57-b855-1a2b8a98670b" containerName="maas-api" Oct 31 17:12:29.256836 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.256804 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.305396 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.305359 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvx92\"/\"openshift-service-ca.crt\"" Oct 31 17:12:29.333009 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.332954 4880 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kvx92\"/\"default-dockercfg-6qmmp\"" Oct 31 17:12:29.333009 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.332991 4880 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvx92\"/\"kube-root-ca.crt\"" Oct 31 17:12:29.351939 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.351898 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-podres\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.351939 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.351987 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-proc\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.351939 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.352013 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-lib-modules\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.351939 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.352039 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-sys\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.351939 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.352075 4880 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcd6w\" (UniqueName: \"kubernetes.io/projected/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-kube-api-access-lcd6w\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.356681 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.356651 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s"] Oct 31 17:12:29.452718 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452666 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-podres\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.452923 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452758 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-proc\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.452923 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452789 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-lib-modules\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.452923 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452808 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-podres\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.452923 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452816 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-sys\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.452923 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452864 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-sys\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.452923 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452883 4880 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcd6w\" (UniqueName: \"kubernetes.io/projected/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-kube-api-access-lcd6w\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.452923 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.452906 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-lib-modules\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.453542 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.453013 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-proc\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.506094 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.506012 4880 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcd6w\" (UniqueName: \"kubernetes.io/projected/5c3699ad-f22f-4465-b7d8-7da49cf80fb2-kube-api-access-lcd6w\") pod \"perf-node-gather-daemonset-jkt9s\" (UID: \"5c3699ad-f22f-4465-b7d8-7da49cf80fb2\") " pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.547199 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.547121 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6796fd4df4-27p22_61704b91-19e2-4915-9f4f-2a1dbf0307c6/authorino/0.log" Oct 31 17:12:29.572486 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.572457 4880 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:29.869620 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.869538 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qmlh_cb9fb5ff-4889-4cf7-aa3a-37a80b636932/manager/1.log" Oct 31 17:12:29.872106 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.872073 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8qmlh_cb9fb5ff-4889-4cf7-aa3a-37a80b636932/manager/0.log" Oct 31 17:12:29.941771 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:29.941737 4880 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s"] Oct 31 17:12:29.948949 ip-10-0-142-127 kubenswrapper[4880]: W1031 17:12:29.948917 4880 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c3699ad_f22f_4465_b7d8_7da49cf80fb2.slice/crio-2a44672b99dbf0b9c19a54a4a2912ae32842a1445497c74f91e0da402ac3f117 WatchSource:0}: Error finding container 2a44672b99dbf0b9c19a54a4a2912ae32842a1445497c74f91e0da402ac3f117: Status 404 returned error can't find the container with id 2a44672b99dbf0b9c19a54a4a2912ae32842a1445497c74f91e0da402ac3f117 Oct 31 17:12:30.823538 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:30.823499 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" event={"ID":"5c3699ad-f22f-4465-b7d8-7da49cf80fb2","Type":"ContainerStarted","Data":"a26b0e2ed1e11665ef782b25eb53f7ebca18bcca00b6aaf04850c4fd3ac8a80f"} Oct 31 17:12:30.823971 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:30.823555 4880 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" event={"ID":"5c3699ad-f22f-4465-b7d8-7da49cf80fb2","Type":"ContainerStarted","Data":"2a44672b99dbf0b9c19a54a4a2912ae32842a1445497c74f91e0da402ac3f117"} Oct 31 17:12:30.823971 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:30.823702 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:30.929543 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:30.929487 4880 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" podStartSLOduration=1.929469907 podStartE2EDuration="1.929469907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 17:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 17:12:30.927555079 +0000 UTC m=+4197.962849884" watchObservedRunningTime="2026-10-31 17:12:30.929469907 +0000 UTC m=+4197.964764708" Oct 31 17:12:32.001836 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:32.001793 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f5vdt_ecf4590e-572d-4c90-ad77-e130e762b449/manager/0.log" Oct 31 17:12:33.769395 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:33.769366 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 17:12:33.777408 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:33.777385 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 17:12:33.788281 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:33.788256 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tgrqs_874393ca-7c09-42ce-8149-d778433b0ea6/extract/0.log" Oct 31 17:12:33.788737 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:33.788718 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 17:12:33.796384 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:33.796354 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7mgwm_1cd2c90f-11c8-46ce-bc99-7bcc746f69f6/extract/0.log" Oct 31 17:12:33.806858 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:33.806833 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-bthm6_e9cad0b0-2b17-48f6-9c14-12a419791b1f/console-operator/3.log" Oct 31 17:12:33.886286 ip-10-0-142-127 kubenswrapper[4880]: E1031 17:12:33.886222 4880 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vs98d" containerName="extractor" Oct 31 17:12:36.840262 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:36.840236 4880 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kvx92/perf-node-gather-daemonset-jkt9s" Oct 31 17:12:37.492773 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:37.492737 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-kns59_4fa7d16a-1eba-4ab3-88f6-d6ae9c638122/cluster-monitoring-operator/0.log" Oct 31 17:12:42.612896 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:42.612869 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qv4ch_49f7f8e5-de0c-4c95-8c4e-2accedb15997/node-exporter/0.log" Oct 31 17:12:43.025949 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:43.025916 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qv4ch_49f7f8e5-de0c-4c95-8c4e-2accedb15997/kube-rbac-proxy/0.log" Oct 31 17:12:43.441519 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:43.441491 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qv4ch_49f7f8e5-de0c-4c95-8c4e-2accedb15997/init-textfile/0.log" Oct 31 17:12:46.741192 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:46.741155 4880 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4wccg_43a99cec-f100-473d-b831-f8052ae72bf0/konnectivity-agent/0.log" limit=1024 Oct 31 17:12:51.845241 ip-10-0-142-127 kubenswrapper[4880]: I1031 17:12:51.845212 4880 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cjljn_b122fe05-653c-4619-a231-c72eefa985b4/istio-proxy/0.log"